1var suggestions=document.getElementById("suggestions"),search=document.getElementById("search");search!==null&&document.addEventListener("keydown",inputFocus);function inputFocus(e){e.ctrlKey&&e.key==="/"&&(e.preventDefault(),search.focus()),e.key==="Escape"&&(search.blur(),suggestions.classList.add("d-none"))}document.addEventListener("click",function(e){var t=suggestions.contains(e.target);t||suggestions.classList.add("d-none")}),document.addEventListener("keydown",suggestionFocus);function suggestionFocus(n){const s=suggestions.classList.contains("d-none");if(s)return;const e=[...suggestions.querySelectorAll("a")];if(e.length===0)return;const t=e.indexOf(document.activeElement);if(n.key==="ArrowUp"){n.preventDefault();const s=t>0?t-1:0;e[s].focus()}else if(n.key==="ArrowDown"){n.preventDefault();const s=t+1<e.length?t+1:t;e[s].focus()}}(function(){var e=new FlexSearch.Document({tokenize:"forward",cache:100,document:{id:"id",store:["href","title","description"],index:["title","description","content"]}});e.add({id:0,href:"/docs/agent/intro/",title:"Pre explanation",description:"Nightingale is an alarm engine, which does not need to be integrated with the collector, but directly connects to various data sources for query alarms.\nThat is to say, if you have collected various monitoring data and stored it in the time series database, you can configure the time series database as a data source in Nightingale, and Nightingale can directly query the data in the time series database. There is no need to use various collectors mentioned in this chapter.",content:"Nightingale is an alarm engine, which does not need to be integrated with the collector, but directly connects to various data sources for query alarms.\nThat is to say, if you have collected various monitoring data and stored it in the time series database, you can configure the time series database as a data source in Nightingale, and Nightingale can directly query the data in the time series database. There is no need to use various collectors mentioned in this chapter.\nHowever, many new users have not built their own collection capabilities, so we provide some collector docking solutions to facilitate users to get started quickly. However, Nightingale still does not provide storage c
1apabilities. These collectors collect data and push it to Nightingale, and Nightingale then forwards the data to the time series database.\nIn the Nightingale configuration file etc/config.toml, there is a [[Pushgw.Writers]] section, which is used to configure the address of the time series database. After receiving the data, Nightingale forwards the data to these addresses.\n"}),e.add({id:1,href:"/docs/prologue/introduction/",title:"Introduction",description:"Nightingale is an open-source project focused on alerting. Similar to Grafana's data source integration approach, Nightingale also connects with various existing data sources. However, while Grafana focuses on visualization, Nightingale emphasizes alerting engines.",content:"Nightingale is an open-source project focused on alerting. Similar to Grafana\u0026rsquo;s data source integration approach, Nightingale also integrates with various existing data sources. While Grafana focuses on visualization, Nightingale focuses on alerting.\nNightingale can query data from multiple data sources, generate alarm events, and then send alerts via various notification channels. In addition, we also have an event pipeline design, which can perform different processing on alarm events, such as filtering, relabeling, enriching, and so on.\nRepo # Backend: https://github.com/ccfos/nightingale Frontend: https://github.com/n9e/fe Any issues or PRs are welcome!\nWorking logic # Many users have already collected metrics and log data themselves. In this case, they can integrate their storage repositories (like VictoriaMetrics, Elasticsearch, etc.) as data sources in Nightingale. Users can then configure alert rules and notification rules in Nightingale to generate and dispatch alert events.\nNightingale itself does not provide data collection capabilities. We recommend using Categraf as a collector, which can seamlessly integrate with Nightingale.\nCategraf can collect monitoring data from operating systems, network devices, middleware, and databases. It pushes this data to Nightingale (via Prometheus Remote Write protocol), which then forwards the data to time-series databases (like Prometheus, VictoriaMetrics, etc.) and provides alerting and visualization capabilities.\nFor specific edge data centers, where the network link to the central Nightingale server is poor, Nightingale also provides a design for edge data center alerting engine deployment. In this mode, even if the edge and central networks are disconnected, alerting functionality remains unaffected.\n In the above diagram, the network link between data center A and the central data center is good, so the alerting engine is handled by the central Nightingale process. For data center B, where the network link to the central data center is poor, we deploy n9e-edge as the alerting engine to handle data source alerting functionality locally.\n Alerting, Upgrades, and Collaboration # Nightingale focuses on being an alerting engine, responsible for generating alert events and flexibly dispatching them based on rules. It has built-in support for 20 notification channels (like phone calls, SMS, email, DingTalk, Feishu, WeCom, Slack, etc.).\nIf you have more advanced requirements, such as:\n Want to aggregate events from multiple monitoring systems into one platform for unified noise reduction, response handling, and data analysis Want to support team on-call culture, including features like alert claim, escalation (to avoid missing alerts), and collaborative handling Then Nightingale may not be suitable. We recommend using FlashDuty, an on-call product that aggregates alerts from various monitoring systems for unified noise reduction, distribution, and response.\nKey Capabilities # Nightingale supports alert rules, muting rules, subscription rules, and notification rules. It natively integrates 20 notification channels and allows customization of message templates. Nightingale supports event pipelines to process alert events and integrate with third-party systems. For example, it can perform operations like relabeling, filtering, and enriching on events. Nightingale supports the concept of business groups and introduces a permission system to manage various rules in a categorized manner. Many databases and middleware have built-in alert rules that can be directly imported for use, and Prometheus alert rules can also be directly imported. Nightingale supports alert self-healing, which means that after an alert is triggered, a script is automatically executed to perform some predefined logic, such as cleaning up the disk or capturing the on-site situation. Nightingale archives historical alert events and supports multi-dimensional querying and statistics. It allows flexible aggregation and grouping, making it easy to get a clear overview of the distribution of the company\u0026rsquo;s alert events at a glance. Nightingale has built-in metric explanations, dashboards, and alert rules for common operating systems, middleware, and databases. However, these are all contributed by the community, and their overall quality varies. Nightingale directly receives data from multiple protocols such as Remote Write, OpenTSDB, Datadog, and Falcon, thus enabling integration with various types of agents. Nightingale supports multiple data sources including Prometheus, ElasticSearch, Loki, and TDEngine, and can perform alerting based on the data from them. Nightingale can be easily embedded into internal enterprise systems, such as Grafana and CMDB. It even allows configuring the menu visibility of these embedded systems. Nightingale supports dashboard functionality, featuring common chart types and some built-in dashboards. The image above is a screenshot of one of these dashboards. If you\u0026rsquo;re already accustomed to Grafana, it\u0026rsquo;s recommended to continue using Grafana for viewing charts, as Grafana has more profound expertise in this area. For machine-related monitoring data collected by Categraf, it\u
10026rsquo;s advisable to use Nightingale\u0026rsquo;s built-in dashboards. This is because Categraf\u0026rsquo;s metric naming follows Telegraf\u0026rsquo;s naming convention, which differs from that of Node Exporter. Since Nightingale incorporates the concept of business groups, where machines can belong to different business groups, there are times when you may only want to view machines belonging to the current business group in the dashboard. Therefore, Nightingale\u0026rsquo;s dashboards can be linked with business groups. Thank you for the trust from numerous enterprises. # Nightingale has been adopted by many enterprises, including but not limited to:\nOpen Source License # The Nightingale monitoring project is open-sourced under the Apache License 2.0.\n"}),e.add({id:2,href:"/docs/prologue/prometheus/",title:"Nightingale vs Prometheus",description:"Nightingale and Prometheus are often discussed in relation to each other, and in fact, they have a complementary relationship. This article will detail the differences and connections between the two.",content:"Nightingale is similar to Grafana in that it can integrate with a variety of data sources, the most common of which is Prometheus-type. Other data sources that are compatible with the Prometheus interface, such as VictoriaMetrics, Thanos, and M3DB, can also be c
1onsidered Prometheus-type sources, so the relationship between the two is close.\nIf you have the following requirements, you might consider using Nightingale:\n You have multiple time-series databases, such as Prometheus and VictoriaMetrics, and want to use a unified platform to manage various alert rules with permission control. You are concerned about the single point of failure of Prometheus\u0026rsquo;s alerting engine and want to avoid downtime. In addition to Prometheus alerts, you need alerts from other data sources such as ElasticSearch, Loki, and ClickHouse. You require more flexible alert rule configurations, such as controlling the effective time, event relabeling, event linkage with CMDB, and supporting alert self-healing scripts. Nightingale also has visualization capabilities similar to Grafana, but it may not be as advanced. In my observation, many companies adopt a combination approach (in the adult world, there are no absolutes):\n Data Collection: A combination of various agents and exporters is used, with Categraf being the primary choice (especially for machine monitoring, seamlessly integrated with Nightingale), supplemented by various exporters. Storage: The time-series database primarily used is VictoriaMetrics, as it is compatible with Prometheus, offers better performance, and has a clustered version. For most companies, the single-node version is sufficient. Alerting Engine: Nightingale is used for alerting, making it easy for different teams to manage and collaborate. It comes with some built-in rules out of the box, and the configuration of alert rules is very flexible, with an event pipeline mechanism that facilitates integration with their own CMDB, etc. Visualization: Grafana is used for visualization, as it offers more advanced and visually appealing charts. The community is also very large, and many pre-made dashboards can be found on the Grafana site, making it relatively hassle-free. On-call Distribution of Alert Events: FlashDuty is used, which supports integration with various monitoring systems such as Zabbix, Prometheus, Nightingale, cloud monitoring solutions, Elastalert, etc. It consolidates alert events into a single platform for unified noise reduction, scheduling, claim escalation, response, distribution, and more. "}),e.add({id:3,href:"/docs/prologue/pre-knowledge/",title:"Prior knowledge",description:"Nightingale monitoring (Nightingale) is an open-source monitoring system. This article introduces some basic knowledge and concepts that need to be understood before learning Nightingale. Monitoring-related knowledge is very complex, and I hope readers can be patient.",content:"Nightingale monitoring (Nightingale) is part of the Prometheus ecosystem, so many concepts and knowledge of Prometheus are prerequisites for using Nightingale. This article lists the key knowledge and provides relevant learning materials, hoping to be helpful to you.\nBasic Knowledge # Linux knowledge, such as process-related, network-related, systemd-related, etc. Prometheus knowledge, which is essential for using Nightingale. You can refer to the Prometheus official documentation. "}),e.add({id:4,href:"/docs/prologue/architecture/",title:"Architecture",description:"This article explains the architecture design of Nightingale monitoring (Nightingale), including the design of centralized clusters and the deployment mode of edge data centers.",content:"Nightingale\u0026rsquo;s architecture is relatively simple. For testing purposes, a single binary can be used to start it. However, for production environments, it relies on MySQL and Redis. Some companies have multiple data centers, and some edge data centers have poor network quality; Nightingale has been specifically designed to address such scenarios.\nArchitecture Diagram # If we do not consider the edge mode, Nightingale has only one main process, the n9e process, which relies on MySQL and Redis to store some management data. It can connect to various data sources, and the technical architecture diagram is as follows:\nInitially supported data sources include Prometheus, VictoriaMetrics, and ElasticSearch, which support both visualization and alerting. However, Nightingale is focusing more on the alert
1ing engine, so new data sources will only support alerting.\nBased on whether the monitoring data flows through Nightingale, we can distinguish two modes:\n Mode 1: Monitoring data does not flow through Nightingale. Users handle data collection themselves and only configure the time-series databases in Nightingale for visualization and alerting. The architecture diagram above is a typical example of this mode. Mode 2: Data flows through Nightingale. Categraf pushes data to Nightingale via the remote write protocol. Nightingale does not store data directly but forwards it to time-series databases, which are determined by the Pushgw.Writers configuration in the config.toml file. The architecture diagram for Mode 2 is as follows: In the diagram above, Nightingale receives monitoring data and forwards it to VictoriaMetrics. It can also forward data to Prometheus; if you want to forward to Prometheus, remember to enable the remote receiver feature when starting Prometheus (you can check which control parameter to use with ./prometheus --help | grep receiver), which enables the /api/v1/write interface in Prometheus.\n ð¢ If you are a new user, it is recommended to use VictoriaMetrics directly, as it offers better performance and supports clustered modes. Additionally, it is compatible with the Prometheus interface. However, there is less Chinese documentation available for VictoriaMetrics compared to Prometheus.\n Single Node Testing Mode # To quickly test Nightingale, download the release package from Nightingale GitHub releases. After downloading, extract the package, and you will find a binary file named n9e. You can run it directly with ./n9e, which defaults to port 17000. The default username is root, and the password is root.2020.\nThe n9e process only relies on the etc and integrations directories in the same directory as the binary and does not depend on any other services.\nIn this single-node mode, it is convenient for quick testing but not recommended for production environments. In this mode, Nightingale stores configuration data (such as user information, alert rules, dashboards, etc.) in a local SQLite database file, so after starting the n9e process, a file named n9e.db will be generated in the same directory.\nSingle Node Production Mode # If you want to deploy Nightingale in a production environment, it requires dependencies on MySQL and Redis. Therefore, you need to configure the connection information for MySQL and Redis in the etc/config.toml file.\nMySQL key configuration example:\n[DB] DBType = \u0026quot;mysql\u0026quot; DSN = \u0026quot;root:YourPa55word@tcp(localhost:3306)/n9e_v6?charset=utf8mb4\u0026amp;parseTime=True\u0026amp;loc=Local\u0026quot; In the above DSN (Data Source Name), the format is username:password@tcp(address:port)/database_name?parameters, where n9e_v6 is the name of the Nightingale database. Since version V6, this name has been consistently used (even though we are now at V8+), and it has been retained in the table creation statements.\nRedis key configuration example:\n[Redis] Address = \u0026quot;127.0.0.1:6379\u0026quot; RedisType = \u0026quot;standalone\u0026quot; The above is just a basic configuration example. The configuration file contains many other configuration items, which can be referenced in the comments within the file or in the configuration file documentation.\nNightingale Cluster # The cluster mode is straightforward; you just need to set up multiple machines, deploy the n9e process on each machine (the process requires the etc and integrations directories to function properly), and ensure that the configuration files of all n9e processes are identical. They should share the same MySQL and Redis instances to work correctly.\nMultiple n9e processes will automatically distribute alert rules. For example, if there are 2 n9e processes and users have configured a total of 100 alert rules, Nightingale will automatically assign these 100 alert rules to the 2 n9e processes, with each process handling approximately 50 alert rules (each rule will only run on one n9e instance, ensuring no duplicate alerts). If one machine goes down, the n9e process on the other machine will take over its alert rules and continue operating.\nEdge Mode # The several modes mentioned above are all centralized modes. However, in actual production environments, there may be multiple computer rooms, and the network quality between the central computer room and edge computer rooms may be poor. If the n9e in the central computer room is made responsible for alerting on a time-series database in an edge computer room, it will be unstable, and sometimes n9e may even fail to connect directly to the time-series database in the edge computer room. In such cases, Nightingale\u
10026rsquo;s edge mode is required.\nAssume your company has 3 data centers: a central primary data center, edge data center A, and edge data center B. A has a dedicated line to the central facility with good network quality, while B uses public network with unreliable connectivity.\nThe n9e process is deployed in the central data center, along with its dependencies MySQL and Redis. For high availability, deploy multiple central n9e instances with identical configurations connecting to the same MySQL and Redis.\nIn the diagram, there are 5 data sources:\n Central data center: Loki and ElasticSearch Edge A: ElasticSearch and Prometheus Edge B: VictoriaMetrics To uniformly view data from all 5 sources in the central n9e, configure their access addresses in Nightingale via Integrations - Data sources.\nThe central n9e can directly access data sources in the central and edge A facilities via internal addresses but cannot reach edge B\u0026rsquo;s VictoriaMetrics (no dedicated line). Thus, expose edge B\u0026rsquo;s VictoriaMetrics via a public address for central n9e access:\n Expose edge B\u0026rsquo;s VictoriaMetrics at a public address (e.g., https://ex.a.com) Configure this public URL in Nightingale\u0026rsquo;s web UI for edge B\u0026rsquo;s VictoriaMetrics Lines 1-5 in the diagram represent connections between the central n9e and the 5 data sources. When users query data via Nightingale\u0026rsquo;s web interface, requests are proxied through the n9e process to backend data sources, which return results to the user.\nn9e-edge is deployed in edge B to handle alert evaluations for its VictoriaMetrics. It synchronizes alert rules from the central n9e (line A in the diagram), caches them in memory, and evaluates alerts against the local VictoriaMetrics. This ensures reliable alerting as n9e-edge and VictoriaMetrics are internally connected. Even if n9e-edge loses connectivity to the central n9e, alert evaluation continues using cached rules.\nAlert events generated by n9e-edge are written to the central MySQL via n9e\u0026rsquo;s API and sent to notification services (DingTalk, Feishu, FlashDuty, etc.). If network connectivity between n9e-edge and n9e fails, events cannot be written to MySQL, but notifications can still be sent if the edge facility has external network access.\nIn the architecture:\n Central n9e handles alert evaluation for central Loki/ElasticSearch and edge A\u0026rsquo;s ElasticSearch/Prometheus Edge B\u0026rsquo;s n9e-edge handles alert evaluation for its VictoriaMetrics To associate data sources with specific alert engines, configure this in the data source management page:\nIn the diagram:\n URL: Address used by central n9e to read data (public address for edge B\u0026rsquo;s VictoriaMetrics in the example) Time-series Database Intranet Address: Address used by n9e-edge to connect to VictoriaMetrics. Leave blank to use the URL if it\u0026rsquo;s already an internal address. For edge B, configure an internal address for reliable alert evaluation. Remote Write URL: VictoriaMetrics\u0026rsquo; remote write address for recording rules. n9e-edge processes recording rules and writes results back to the time-series database, so use an internal address. Optional if recording rules are unused. Associated Alert Engine Cluster: Selected as \u0026ldquo;edge-b\u0026rdquo; (name specified by EngineName in edge.toml), establishing the association between edge B\u0026rsquo;s n9e-edge and its VictoriaMetrics for handling alert/recording rules. Newer Nightingale versions require n9e-edge to use a dedicated Redis instance in edge B, separate from the central n9e\u0026rsquo;s Redis (labeled R1 and R2 in the diagram).\nFor Categraf: if network connectivity is good (central and edge A), Categraf can report directly to the central n9e. For edge B with n9e-edge, configure Categraf to connect to the local n9e-edge.\nConfiguration Examples # Sample configurations for the above architecture:\nCentral n9e Configuration # The central n9e uses etc/config.toml:\n[HTTP.APIForService] Enable = true [HTTP.APIForService.BasicAuth] user001 = \u0026quot;ccc26da7b9aba533cbb263a36c07dcc5\u0026quot; user002 = \u0026quot;ccc26da7b9aba533cbb263a36c07dcc6\u0026quot; Key configuration: HTTP.APIForService. Default Enable = false (security measure). Set to true to supp
1ort n9e-edge. n9e-edge uses BasicAuth for authentication (configured above with two users for demonstration). If n9e is exposed publicly, modify default passwords to prevent attacks.\nEdge n9e-edge Configuration # n9e-edge uses etc/edge/edge.toml. Configure the central n9e address for rule synchronization:\n[CenterApi] Addrs = [\u0026quot;http://N9E-CENTER-SERVER:17000\u0026quot;] BasicAuthUser = \u0026quot;user001\u0026quot; BasicAuthPass = \u0026quot;ccc26da7b9aba533cbb263a36c07dcc5\u0026quot; # unit: ms Timeout = 9000 N9E-CENTER-SERVER:17000 is the central n9e address. BasicAuthUser and BasicAuthPass match central credentials (omit if BasicAuth is disabled). Always modify default passwords for public exposure.\nNew n9e-edge versions require Redis (configure in edge.toml). Check for Redis settings in the default edge.toml to confirm dependency.\nEdge Categraf Configuration # Configure writer and heartbeat addresses to point to n9e-edge:\n... [[writers]] url = \u0026quot;http://N9E-EDGE:19000/prometheus/v1/write\u0026quot; ... [heartbeat] enable = true # report os version cpu.util mem.util metadata url = \u0026quot;http://N9E-EDGE:19000/v1/n9e/heartbeat\u0026quot; ... N9E-EDGE:19000 is the n9e-edge address (default port 19000, configurable in edge.toml).\nIbex Configuration # For self-healing functionality (ibex), enable in edge.toml:\n[Ibex] Enable = true RPCListen = \u0026quot;0.0.0.0:20090\u0026quot; Configure edge Categraf to connect to n9e-edge\u0026rsquo;s RPC port:\n[ibex] enable = true ## ibex flush interval interval = \u0026quot;1000ms\u0026quot; ## n9e ibex server rpc address servers = [\u0026quot;N9E-EDGE-IP:20090\u0026quot;] ## temp script dir meta_dir = \u0026quot;./meta\u0026quot; N9E-EDGE-IP:20090 is the n9e-edge RPC address (no http:// prefix for RPC).\nOther Use Cases # Beyond poor network links, edge mode suits secure network partitions where only a relay machine connects to the central n9e. Deploy n9e-edge on the relay, with other machines\u0026rsquo; Categraf instances connecting to it.\n"}),e.add({id:5,href:"/docs/prologue/flashcat/",title:"Enterprise Edition",description:"The monitoring field is indeed very complex. If you need third-party assistance, you can consider the Nightingale commercial version, which offers more features and services.",content:"Nightingale\u0026rsquo;s core development team is fully dedicated to open source. Without a sustainable income, the project cannot continue. Therefore, they founded a company: Beijing Flashcat Cloud Technology Co., Ltd., abbreviated as \u0026ldquo;Flashcat\u0026rdquo; and launched two commercial products:\n One-stop Intelligent Observation Platform Flashcat One-stop Alarm Response Platform Flashduty Among them, Flashcat can be regarded as the commercial version of Nightingale, positioned as a unified observation platform, while the open-source version focuses only on metric monitoring. The differences between the two can be referenced in this article: ãDifferences Between Nightingale Open Source Version and Commercial Versionã.\nFlashDuty is a one-stop alarm response platform similar to PagerDuty (i.e., On-call center), supporting integration with various monitoring systems such as Zabbix, Prometheus, Nightingale, cloud monitoring, Elastalert, etc. It consolidates alarm events into a single platform and unifies the processing of alarms, including convergence and noise reduction, scheduling, claim escalation, response, dispatch, and more. You can register for a free trial .\nThe knowledge in the monitoring and observability field is indeed very complex. If you have the budget and need third-party assistance, you can consider Flashcat\u0026rsquo;s products for a win-win situation ð Free product idea exchange with Flashcat.\n"}),e.add({id:6,href:"/docs/prologue/videos/",title:"Videos for Getting Started",description:"Nightingale's introductory video tutorials help users quickly get started with Nightingale.",content:"To help users quickly get started with Nightingale, we have created a series of introductory video tutorials covering various aspects from installation and deployment to basic usage. Here is the list of these videos (it is recommended to watch at double speed for a quick overview):\n Nightingale Project Introduction, Resource Links, etc., Must-Watch Nightingale Architecture Explanation Basic Functionality Introduction Installing Categraf for Nightingale Integration Introduction to Common Categraf Plugins Notification Rules Introduction "}),e.add({id:7,href:"/docs/install/pre-intro/",title:"Pre-Introduction",description:"Nightingale supports various installation methods, including binary deployment, Docker Compose deployment, and Helm deployment. Which one should you choose? This article provides some suggestions.",content:"Common installation methods include:\n Binary deployment Docker Compose deployment Helm deployment The recommended method is binary deployment, for the following reasons:\n Nightingale consists of a single binary file with minimal dependencies, making it easier to manage. Most users are familiar with systemd, so you can simply use systemd to manage the Nightingale process. Docker Compose deployment may have slightly lower performance compared to binary deployment, and it requires additional knowledge of Docker. Additionally, issues with pulling images due to network restrictions in China can be troublesome. Helm deployment is suitable for Kubernetes environments, but since monitoring systems are critical (P0 level), if Kubernetes goes down, the monitoring system will also be affected. This can lead to complaints from other teams about the lack of planning. Regardless of the installation method, after installation, the default username for Nightingale is root, and the password is root.2020. Nightingale listens on port 17000 by default, and in edge mode, the n9e-edge port is 19000.\nIf you are using edge mode, please be sure to read the Edge Mode Documentation section.\n"}),e.add({id:8,href:"/docs/install/upgrade/",title:"Upgrade",description:"Nightingale monitoring (Nightingale) how to upgrade between different versions, what to pay attention to, and which files to replace",content:"Nightingale versions V6, V7, and V8 have the same upgrade method between their minor versions.\nUpgrade Steps # Backup Data: Before upgrading, back up the MySQL database content, binary files, and the etc and integrations directories. Having a backup allows you to operate confidently. If you are using binary deployment, replace the binary files and the integrations directory (you can simply move the old integrations directory to a backup location using mv integrations integrations.bak, and then use the new integrations directory). Compare the new and old configuration files using diff, and manually fill in any differences (in practice, you should rarely need to modify the configuration files, as they have not changed much for a long time). If you are using container deployment, pull the latest image, compare the configuration files, fill in any differences, and then restart the container. About Database Table Structure # If the database account used by Nightingale has permissions to create and modify tables, you do not need to manually change the database table structure. Nightingale will automatically check if the table structure needs to be upgraded at startup, and if so, it will modify the tables automatically. If the database account does not have permissions to create or modify tables, you will need to make manual adjustments. Recent changes can be found in migrate.sql. If automatic table modification fails, please submit an issue, and we will follow up as soon as possible.\nIn theory, the database supports both MySQL and Postgres, but the community lacks long-term contributors for Postgres, so it is recommended to use MySQL first.\n"}),e.add({id:9,href:"/docs/install/binary/",title:"Binary Deployment",description:"Deploying Nightingale Monitoring using binary files",content:"If you have not read the section on Pre-Installation Instru
1ctions, please do so before reading this section.\nDownload # Download the latest version from GitHub, and you will get a compressed package similar to n9e-${version}-linux-amd64.tar.gz. This is the release package for the X86 CPU architecture. If you need the ARM architecture, download the arm64 package. There is no Windows version of the release package because Nightingale is a server-side project that typically runs on Linux systems.\nIf you want to run Nightingale on Windows and Mac, it\u0026rsquo;s also OK, but you need to compile it yourself. The compilation is relatively simple, and you can refer to the logic in the Makefile in the project code repository.\nUnzip the downloaded package to the /opt/n9e directory.\nmkdir /opt/n9e \u0026amp;\u0026amp; tar zxvf n9e-${version}-linux-amd64.tar.gz -C /opt/n9e Single Node Test Installation # In this mode, it is only for testing purposes, and it does not depend on MySQL or Redis (it actually uses SQLite and an in-memory Redis: miniredis). The startup is straightforward; you can start it directly after unzipping.\nStart the Process # cd /opt/n9e \u0026amp;\u0026amp; nohup ./n9e \u0026amp;\u0026gt; n9e.log \u0026amp; Since this is just a test mode, we directly use nohup to start it. In a production environment, you would typically use systemd to manage the n9e process.\nCheck the Process # # Check if the process is running ss -tlnp | grep 17000 Login # Open your browser and visit http://localhost:17000. The default username is root, and the default password is root.2020.\n Please replace localhost with your server\u0026rsquo;s IP address.\n Single Node Production Installation # In a production environment, we recommend using MySQL and Redis to store data.\nModify Configuration # You need to configure the connection information for MySQL and Redis in the /opt/n9e/etc/config.toml file.\nMySQL key configuration example:\n[DB] DBType = \u0026quot;mysql\u0026quot; DSN = \u0026quot;YourUsername:YourPassword@tcp(127.0.0.1:3306)/n9e_v6?charset=utf8mb4\u0026amp;parseTime=True\u0026amp;loc=Local\u0026quot; Redis key configuration example:\n[Redis] Address = \u0026quot;127.0.0.1:6379\u0026quot; Password = \u0026quot;YourRedisPassword\u0026quot; RedisType = \u0026quot;standalone\u0026quot; Start the Process # Start the n9e binary, and Nightingale will automatically create the database tables. This requires your database connection account to have permissions to create and modify tables.\nnohup ./n9e \u0026amp;\u0026gt; n9e.log \u0026amp; # Check if the process started successfully ps -ef | grep n9e ss -tlnp | grep 17000 nohup allows for quick startup verification; if there are any issues, check the n9e.log file.\nUsing systemd for Management # In a production environment, it is recommended to use systemd to manage the n9e process. Create a systemd service file at /etc/systemd/system/n9e.service with the following content:\n[Unit] Description=Nightingale Monitoring Service After=network.target [Service] Type=simple ExecStart=/opt/n9e/n9e WorkingDirectory=/opt/n9e Restart=always RestartSec=5 StandardOutput=syslog StandardError=syslog SyslogIdentifier=n9e [Install] WantedBy=multi-user.target After saving the file, run the following commands to enable and start the service:\nsudo systemctl enable n9e sudo systemctl start n9e Login # Open your browser and visit http://localhost:17000. The default username is root, and the default password is root.2020.\n Please replace localhost with your server\u0026rsquo;s IP address.\n Cluster Mode # The logic of the cluster mode has already been explained in Nightingale Architecture Design, so it will not be repeated here. From the deployment perspective, you just need to set up multiple machines, deploy one n9e process on each machine, and configure the connection information for MySQL and Redis properly. Multiple n9e processes share the same set of MySQL and Redis, so the configuration files of these n9e processes are exactly the same.\nEdge Mode # For instructions on the edge mode, please be sure to read this first: Nightingale Monitoring - Detailed Explanation of Edge Alert Engine Architecture!!!\nThe edge mode uses the n9e-edge binary, which can be found in the n9e-${version}-linux-amd64.tar.gz compressed package. n9e-edge needs to communicate with the n9e of the central server to synchronize alert rules, so the configuration file of n9e-edge must include the connection information of the central n9e.\nEdge Cluster # Multiple instances of n9e-edge in edge computer rooms can also be deployed to form a cluster. The configuration files of multiple n9e-edge instances within the same cluster must be consistent. Instances with the same EngineName in the configuration file will be regarded as multiple instances of the same engine cluster, and the name should be different from that of the central n9e. The default EngineName of the central n9e is default, while the default EngineName of the edge n9e-edge is edge.\nIf you have multiple edge computer rooms, the EngineName of n9e-edge in each edge computer room needs to be different, such as edge1, edge2, etc. Only after such differentiation can different data sources be assigned to different alert engines.\nStart n9e-edge # Note that when starting the n9e-edge process, you need to specify the configuration directory instead of the configuration file. For example:\nnohup ./n9e-edge --configs etc/edge \u0026amp;\u0026gt; edge.log \u0026amp; The etc/edge mentioned above is the configuration directory. It would be incorrect to write it as --configs etc/edge/edge.toml.\n"}),e.add({id:10,href:"/docs/install/compose/",title:"Docker Compose",description:"Nightingale monitoring (Nightingale) supports Docker Compose deployment, this article introduces how to deploy Nightingale using Docker Compose.",content:"Download # Refer to the Binary Installation section to download the Nightingale monitoring release package, which includes Docker Compose configuration files. Alternatively, you can directly download the Nightingale source repository, where you can also find the Docker Compose configuration files.\nStart # Whether you download the release package or the source code repository, there will be a docker/compose-bridge directory after decompression. Simply enter this directory and execute the docker-compose up -d command (due to domestic network restrictions, image downloads may fail, and you need to solve the problem of scientific internet access on your own).\nroot@ubuntu-linux-22-04-desktop:/opt/n9e/docker/compose-bridge# docker
1compose up -d [+] Running 5/5 â Container victoriametrics Started 0.6s â Container redis Started 0.6s â Container mysql Started 0.6s â Container nightingale Started 0.2s â Container categraf Started 0.2s root@ubuntu-linux-22-04-desktop:/opt/n9e/docker/compose-bridge# docker compose ps NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS categraf m.daocloud.io/docker.io/flashcatcloud/categraf:latest \u0026quot;/entrypoint.sh\u0026quot; categraf 2 minutes ago Up 3 seconds mysql mysql:8 \u0026quot;docker-entrypoint.sâ¦\u0026quot; mysql 2 minutes ago Up 4 seconds 0.0.0.0:3306-\u0026gt;3306/tcp, :::3306-\u0026gt;3306/tcp, 33060/tcp nightingale m.daocloud.io/docker.io/flashcatcloud/nightingale:latest \u0026quot;sh -c /app/n9e\u0026quot; nightingale 2 minutes ago Up 3 seconds 0.0.0.0:17000-\u0026gt;17000/tcp, :::17000-\u0026gt;17000/tcp, 0.0.0.0:20090-\u0026gt;20090/tcp, :::20090-\u0026gt;20090/tcp redis redis:6.2 \u0026quot;docker-entrypoint.sâ¦\u0026quot; redis 2 minutes ago Up 4 seconds 0.0.0.0:6379-\u0026gt;6379/tcp, :::6379-\u0026gt;6379/tcp victoriametrics victoriametrics/victoria-metrics:v1.79.12 \u0026quot;/victoria-metrics-pâ¦\u0026quot; victoriametrics 2 minutes ago Up 4 seconds 0.0.0.0:8428-\u0026gt;8428/tcp, :::8428-\u0026gt;8428/tcp Docker Compose starts multiple containers, which are:\n victoriametrics: Time-series database, compatible with Prometheus, with better performance. redis: Cache database, Nightingale uses Redis to store JWT tokens and machine heartbeat metadata mysql: Relational database, Nightingale uses MySQL to store user information, alert rules, dashboards, and other configuration data. nightingale: The core service of Nightingale monitoring. categraf: Monitoring data collector, responsible for collecting CPU, memory, disk, and other metric data from the host. Login # Use your browser to visit http://localhost:17000 to open the Nightingale monitoring page. The default username is root, and the default password is root.2020.\n Please replace localhost with your server\u0026rsquo;s IP address.\n Cluster Mode # In cluster mode, multiple n9e instances need to share the same MySQL and Redis, so you cannot simply use the default Docker Compose. You need to modify the config.toml file in the etc-nightingale directory to configure the unified MySQL and Redis connection information.\nEdge Mode # The edge mode requires the n9e-edge process. However, the community does not provide a Docker image for n9e-edge, so in edge mode, it is still necessary to deploy the n9e-edge process using the binary method. For detailed instructions on the edge mode, please refer to: Nightingale Monitoring - Detailed Explanation of Edge Alert Engine Architecture.\n"}),e.add({id:11,href:"/docs/install/helm/",title:"Helm",description:"Use Helm chart to install Nightingale monitoring (Nightingale) and deploy it in Kubernetes.",content:"You can run Nightingale using the n9e helm chart to deploy it in a Kubernetes cluster.\nThe default username for Nightingale is root, and the password is root.2020.\nHowever, we do not recommend deploying Nightingale in Kubernetes because the monitoring system is too critical. If the Kubernetes cluster encounters issues, it may cause the monitoring system to malfunction. At that time, you might want to use the monitoring data to troubleshoot Kubernetes issues, leading to a circular dependency. Especially when other teams find that they cannot use the monitoring system, they may come to you with complaints.\n"}),e.add({id:12,href:"/docs/install/configuration/",title:"Configuration",description:"Explanation of the configuration files for Nightingale monitoring, with detailed descriptions of each configuration item",content:"The configuration file for the central n9e is etc/config.toml, and the configuration file for the edge alert engine n9e-edge is etc/edge/edge.toml. Here we first explain the n9e configuration file in sections.\nGlobal # [Global] RunMode = \u0026quot;release\u0026quot; This is a configuration item used by Nightingale developers; ordinary users don\u0026rsquo;t need to care about it. It should always remain release.\nLog # [Log] # stdout, stderr, file Output = \u0026quot;stdout\u0026quot; # log write dir Dir = \u0026quot;logs\u0026quot; # log level: DEBUG INFO WARNING ERROR Level = \u0026quot;DEBUG\u0026quot; # # rotate by time # KeepHours = 4 # # rotate by size # RotateNum = 3 # # unit: MB # RotateSize = 256 Output: Log output method, supporting stdout, stderr, file. Only in file mode will logs be output to files, and the following configuration items will be used. Dir: Directory for storing log files Level: Log level, supporting DEBUG, INFO, WARNING, ERROR KeepHours: Log file retention time in hours. Logs can be rotated by time or size. If rotated by time, use this configuration item (one log file per hour); if rotated by size, use the following two configuration items. RotateNum: Number of retained log files RotateSize: Log file size in MB HTTP # [HTTP] # http listening address Host = \u0026quot;0.0.0.0\u0026quot; # http listening port Port = 17000 # https cert file path CertFile = \u0026quot;\u0026quot; # https key file path KeyFile = \u0026quot;\u0026quot; # whether print access log PrintAccessLog = false # whether enable pprof PProf = true # expose prometheus /metrics? ExposeMetrics = true # http graceful shutdown timeout, unit: s ShutdownTimeout = 30 # max content length: 64M MaxContentLength = 67108864 # http server read timeout, unit: s ReadTimeout = 20 # http server write timeout, unit: s WriteTimeout = 40 # http server idle timeout, unit: s IdleTimeout = 120 Host: HTTP service listening address, usually 0.0.0.0 to listen on all network interfaces Port: HTTP service listening port CertFile: HTTPS certificate file path KeyFile: HTTPS key file path PrintAccessLog: Whether to print access logs PProf: Whether to enable pprof. If enabled, pprof information can be viewed at /api/debug/pprof/ ExposeMetrics: Whether to expose Prometheus\u0026rsquo;
1s /metrics interface for exposing Nightingale\u0026rsquo;s own monitoring metrics ShutdownTimeout: HTTP service graceful shutdown timeout in seconds MaxContentLength: Maximum HTTP request length in bytes ReadTimeout: HTTP read timeout in seconds WriteTimeout: HTTP write timeout in seconds IdleTimeout: HTTP idle timeout in seconds HTTP.ShowCaptcha # [HTTP.ShowCaptcha] Enable = false Enable: Whether to enable the captcha function HTTP.APIForAgent # [HTTP.APIForAgent] Enable = true # [HTTP.APIForAgent.BasicAuth] # user001 = \u0026quot;ccc26da7b9aba533cbb263a36c07dcc5\u0026quot; Enable: Whether to enable the API interface for Agent. Normally, this must be enabled, so this configuration item is generally true. BasicAuth: The Agent\u0026rsquo;s API interface supports BasicAuth. Configure BasicAuth username and password here. For internal network communication, BasicAuth is not required; for public network communication, it is recommended to configure BasicAuth, and the password must not use the default one to avoid attacks. In the example above, user001 is the BasicAuth username, and ccc26da7b9aba533cbb263a36c07dcc5 is the BasicAuth password. To configure multiple users, you can add more entries, for example: [HTTP.APIForAgent.BasicAuth] user001 = \u0026quot;ccc26da7b9aba533cbb263a36c07dcc5\u0026quot; user002 = \u0026quot;d4f5e6a7b8c9d0e1f2g3h4i5j6k7l8m9\u0026quot; Note: If you configure BasicAuth, the corresponding username and password must also be configured in the Agent\u0026rsquo;s n9e configuration file; otherwise, the Agent cannot connect to the central n9e.\nThe default configuration has Enable set to true and HTTP.APIForAgent.BasicAuth empty, indicating that the API interfaces for Agent are enabled without BasicAuth.\nHTTP.APIForService # [HTTP.APIForService] Enable = false [HTTP.APIForService.BasicAuth] user001 = \u0026quot;ccc26da7b9aba533cbb263a36c07dcc5\u0026quot; Enable: Whether to enable the API interface for Service. The edge alert engine n9e-edge communicates with the central n9e through these interfaces. So if you use n9e-edge, this needs to be enabled (set to true). BasicAuth: The Service\u0026rsquo;s API interface supports BasicAuth. Configure BasicAuth username and password here. For internal network communication, BasicAuth is not required; for public network communication, it is recommended to configure BasicAuth, and the password must not use the default one to avoid attacks. In the example above, user001 is the BasicAuth username, and ccc26da7b9aba533cbb263a36c07dcc5 is the BasicAuth password. To configure multiple users, you can add more entries, for example: [HTTP.APIForService.BasicAuth] user001 = \u0026quot;ccc26da7b9aba533cbb263a36c07dcc5\u0026quot; user002 = \u0026quot;d4f5e6a7b8c9d0e1f2g3h4i5j6k7l8m9\u0026quot; Note: If you configure BasicAuth, the corresponding username and password must also be configured in the n9e-edge configuration file; otherwise, n9e-edge cannot connect to the central n9e. The default configuration has Enable set to false, meaning the API interfaces for other Services are not enabled, and n9e-edge cannot connect to the central n9e.\nHTTP.JWTAuth # [HTTP.JWTAuth] # unit: min AccessExpired = 1500 # unit: min RefreshExpired = 10080 RedisKeyPrefix = \u0026quot;/jwt/\u0026quot; Nightingale uses JWT for authentication. Here configure JWT expiration times in minutes: AccessExpired is the expiration time of the access token, and RefreshExpired is the expiration time of the refresh token. For the roles of these two tokens in JWT mechanism, you can ask GPT for details, which will not be elaborated here. Nightingale stores some JWT-related information in Redis, and RedisKeyPrefix is the prefix for Redis keys, which generally does not need to be changed.\nHTTP.ProxyAuth # [HTTP.ProxyAuth] # if proxy auth enabled, jwt auth is disabled Enable = false # username key in http proxy header HeaderUserNameKey = \u0026quot;X-User-Name\u0026quot; DefaultRoles = [\u0026quot;Standard\u0026quot;] If you want to embed Nightingale into your own system, you can consider using ProxyAuth, similar to Grafana\u0026rsquo;s ProxyAuth. It means that after a user logs in to your system, you can get the username and put it in the X-User-Name header to pass to Nightingale, and Nightingale will consider the user logged in. DefaultRoles is the default role; if you don\u0026rsquo;t pass roles, Nightingale will treat the user as having the Standard role.\nIn fact, according to observations, no community users are currently using this function, so please use it with caution.\nHTTP.RSA # [HTTP.RSA] OpenRSA = false When logging in to Nightingale, user passwords are transmitted in plain text. If the Nightingale site uses HTTPS, it\u0026rsquo;s fine; if it\u0026rsquo;s HTTP, it\u0026rsquo;s recommended to enable RSA encryption to prevent plain text transmission of user passwords.\nDB # [DB] # mysql postgres sqlite DBType = \u0026quot;sqlite\u0026quot; # postgres: host=%s port=%s user=%s dbname=%s password=%s sslmode=%s # postgres: DSN=\u0026quot;host=127.0.0.1 port=5432 user=root dbname=n9e_v6 password=1234 sslmode=disable\u0026quot; # mysql: DSN=\u0026quot;root:1234@tcp(localhost:3306)/n9e_v6?charset=utf8mb4\u0026amp;parseTime=True\u0026amp;loc=Local\u0026quot; DSN = \u0026quot;n9e.db\u0026quot; # enable debug mode or not Debug = false # unit: s MaxLifetime = 7200 # max open connections MaxOpenConns = 32 # max idle connections MaxIdleConns = 8 DBType and DSN are the most critical, and the two configurations are linked. DBType supports three databases: mysql, postgres, and sqlite. DSN is the database connection information;
1 for sqlite, it\u0026rsquo;s the database file path; for mysql or postgres, it\u0026rsquo;s the database connection information.\nStarting from version v8, Nightingale sets DBType to sqlite by default to facilitate users\u0026rsquo; quick experience without installing a database. However, in production environments, please use mysql or postgres.\nFor DSN configurations of Postgres and MySQL, you can refer to the examples in the comments. Other configurations are related to database connections and can be modified according to your environment. For general small and medium-sized environments, setting MaxOpenConns to 32 and MaxIdleConns to 8 is sufficient.\nRedis # [Redis] # standalone cluster sentinel miniredis RedisType = \u0026quot;miniredis\u0026quot; # address, ip:port or ip1:port,ip2:port for cluster and sentinel(SentinelAddrs) Address = \u0026quot;127.0.0.1:6379\u0026quot; # Username = \u0026quot;\u0026quot; # Password = \u0026quot;\u0026quot; # DB = 0 # UseTLS = false # TLSMinVersion = \u0026quot;1.2\u0026quot; # Mastername for sentinel type # MasterName = \u0026quot;mymaster\u0026quot; # SentinelUsername = \u0026quot;\u0026quot; # SentinelPassword = \u0026quot;\u0026quot; Redis is used not only to store JWT-related login authentication information but also to store metadata reported by machine heartbeats. The machine offline alert rules supported in Nightingale judge based on the heartbeat time of machines in Redis. If there is no heartbeat for a long time, the machine is considered offline.\n If Redis responds slowly, it may cause false judgments of offline alerts. That is, the machine is actually alive, but the heartbeat information in Redis is not updated in time, eventually leading Nightingale to mistakenly judge the machine as offline. Starting from version V8.beta11, monitoring indicators related to Redis operations have been added. These indicators need to be paid attention to to detect slow Redis response issues in time.\n RedisType supports four types: standalone, cluster, sentinel, and miniredis. Starting from Nightingale v8, Nightingale uses miniredis by default to facilitate users\u0026rsquo; quick experience without installing Redis. However, in production environments, please use other modes.\nAddress is the Redis connection address, and the configuration method varies according to RedisType:\n standalone: When RedisType is standalone, Address is the address of the Redis instance in the format ip:port cluster: When RedisType is cluster, Address is the address of the Redis cluster in the format ip1:port,ip2:port sentinel: When RedisType is sentinel, Address is the address of Redis Sentinel in the format ip1:port,ip2:port. In sentinel mode, MasterName, SentinelUsername, and SentinelPassword also need to be configured UseTLS: Whether to use TLS TLSMinVersion: Minimum TLS version, which takes effect only when UseTLS is true Alert # Starting from a certain version, Nightingale merged the webapi and alert engine modules to reduce deployment complexity. The configuration items here under Alert are for the alert engine.\nAlert.Heartbeat # [Alert.Heartbeat] # auto detect if blank IP = \u0026quot;\u0026quot; # unit ms Interval = 1000 EngineName = \u0026quot;default\u0026quot; IP: The IP address of the alert engine. If empty, Nightingale will automatically detect it. Each alert engine writes heartbeat information to MySQL, so that each alert engine knows the list of all alive alert engines, and then can perform sharding processing of alert rules. For example, with 100 alert rules and a cluster of two n9e instances, each n9e will process approximately 50 rules. When one alert engine goes down, the other will take over all 100 rules. Interval: Heartbeat interval in milliseconds EngineName: The name of the alert engine. Generally, the central end maintains default; for the edge alert engine n9e-edge, you can customize the EngineName, such as edge1, edge2, etc. Alert engines with the same EngineName are considered a cluster. Center # Unique configurations for the central n9e, not present in the edge alert engine n9e-edge. These are the unique configurations related to the old version of n9e-webapi.\n[Center] MetricsYamlFile = \u0026quot;./etc/metrics.yaml\u0026quot; I18NHeaderKey = \u0026quot;X-Language\u0026quot; [Center.AnonymousAccess] PromQuerier = true AlertDetail = true MetricsYamlFile: Path to the metric configuration file. The explanations of metrics you see in the quick view come from this configuration file. Later, the metric view was launched, making this configuration file less important, and even the quick view function is planned to be removed. I18NHeader: This is a configuration item for developers; ordinary users don\u0026rsquo;t need to care about it. Center.AnonymousAccess: Configuration items related to anonymous access. PromQuerier indicates whether to allow anonymous query of interfaces of various data sources; AlertDetail indicates whether to allow anonymous viewing of alert details. It can be enabled in internal network environments but must be disabled in public network environments. The dashboard has a public access function, which can even be set to be accessible without login, but this requires PromQuerier to be set to true. That is, if PromQuerier = false, even if the dashboard is set to public access, login is still required.\nPushgw # Although Nightingale does not directly store monitoring data, it provides multiple interfaces for receiving monitoring data, such as interfaces for the Prometheus remote write protocol, OpenTSDB protocol, etc. After receiving the data, Nightingale forwards the monitoring data to the backend time-series database, so
1Nightingale acts as a Pushgateway here, and the configuration items related to Pushgateway are under Pushgw.\n[Pushgw] # use target labels in database instead of in series LabelRewrite = true ForceUseServerTS = true LabelRewrite: Nightingale has a machine management menu where you can tag machines, and these tags are attached to time-series data related to the machines. However, if a tag in the reported data conflicts with a tag in machine management, which one takes precedence? If LabelRewrite is true, the tag in machine management takes precedence; otherwise, the reported tag takes precedence. ForceUseServerTS: Whether to force the use of the server\u0026rsquo;s timestamp to overwrite the timestamp of the received monitoring data. Previously, there was no this configuration item. Due to confusion caused by uncalibrated machine time in many companies, Nightingale provides this configuration. It is recommended to enable it to uniformly use the server\u0026rsquo;s timestamp. Pushgw.DebugSample # [Pushgw.DebugSample] ident = \u0026quot;xx\u0026quot; __name__ = \u0026quot;cpu_usage_active\u0026quot; This configuration is for debugging and troubleshooting. It is actually a filter condition for monitoring metrics. If the metrics reported to Nightingale meet this filter condition, they will be printed to the log. Generally, no configuration is needed; it can be commented out.\nPushgw.WriterOpt # [Pushgw.WriterOpt] QueueMaxSize = 1000000 QueuePopSize = 1000 QueueNumber = 0 This part of the configuration is commented out by default because normally users don\u0026rsquo;t need to pay attention to it. If Nightingale receives too much data, which gets congested in memory and eventually leads to metric loss, you need to consider adjusting the configuration here.\nNightingale creates QueueNumber queues in memory. After receiving monitoring data, it puts the data into these queues. The default configuration of QueueNumber is 0, indicating that the specific number is not specified, and queues are created according to the number of CPU cores. The maximum capacity of each queue is QueueMaxSize, which defaults to 1000000, meaning each queue can store up to 1 million data entries.\nEach queue corresponds to a goroutine, which fetches QueuePopSize metrics from the queue each time. The default is 1000, meaning 1000 data entries are fetched from the queue each time and written to the backend time-series database as a batch. This takes full advantage of multi-core CPU performance. Therefore, the number of QueueNumber is essentially equal to the concurrency of writing to the backend time-series database.\nPushgw.Writers # Here configure the remote write addresses of the backend time-series databases. All time-series databases supporting the remote write protocol can be configured here. Generally, only one needs to be configured; if you want to write to multiple time-series databases simultaneously, you can configure multiple.\n[[Pushgw.Writers]] Url = \u0026quot;http://127.0.0.1:9090/api/v1/write\u0026quot; BasicAuthUser = \u0026quot;xx\u0026quot; BasicAuthPass = \u0026quot;xx\u0026quot; [[Pushgw.Writers]] Url = \u0026quot;http://127.0.0.1:8482/api/v1/write\u0026quot; BasicAuthUser = \u0026quot;xx\u0026quot; BasicAuthPass = \u0026quot;xx\u0026quot; Url: Remote write address of the time-series database BasicAuth: If the time-series database requires BasicAuth authentication, configure the BasicAuth username and password here Headers: If the time-series database requires additional headers, configure them here Timeout: Write timeout in milliseconds DialTimeout: Connection timeout in milliseconds Pushgw.Writers.WriteRelabels # Data written to the time-series database can undergo relabeling before writing. This configuration item is for relabeling, similar to Prometheus\u0026rsquo;s relabel configuration items, except that Prometheus uses yaml format while Nightingale uses toml format.\nIbex # Configuration items for the fault self-healing engine Ibex, i.e., the function for remote script execution. Originally, this function was a separate module called ibex, which was later merged into n9e, so this configuration item is also in n9e.\n[Ibex] Enable = true RPCListen = \u0026quot;0.0.0.0:20090\u0026quot; Enable: Whether to enable the Ibex server function RPCListen: RPC service listening address of Ibex n9e-edge configuration # The configuration file for the edge alert engine n9e-edge is etc/edge/edge.toml, and most configurations are the same as those of the central n9e. For more information, you can refer to this article: ãNightingale Monitoring - Edge Alert Engine Architecture Detailed Explanationã.\n"}),e.add({id:13,href:"/docs/agent/categraf/",title:"Categraf",description:"Use Categraf as collector for Nightingale",content:"Categraf is an agent which can collect metrics and logs. Categraf uses prometheus remote write as data push protocol, so it can push metrics to Nightingale.\nConfiguration # Configuration file of categraf: conf/config.toml\n[writer_opt] # default: 2000 batch = 2000 # channel(as queue) size chan_size = 10000 [[writers]] url = \u0026quot;http://N9E:17000/prometheus/v1/write\u0026quot; # Basic auth username basic_auth_user = \u0026quot;\u0026quot; # Basic auth password basic_auth_pass = \u0026quot;\u0026quot; # timeout settings, unit: ms timeout = 5000 dial_timeout = 2500 max_idle_conns_per_host = 100 [heartbeat] enable = true # report os version cpu.util mem.util metadata url = \u0026quot;http://N9E:17000/v1/n9e/heartbeat\u0026quot; # interval, unit: s interval = 10 # Basic auth username basic_auth_user = \u0026quot;\u0026quot; # Basic auth password basic_auth_pass = \u0026quot;\u0026quot; ## Optional headers # headers = [\u0026quot;X-From\u0026quot;, \u0026quot;
1categraf\u0026quot;, \u0026quot;X-Xyz\u0026quot;, \u0026quot;abc\u0026quot;] # timeout settings, unit: ms timeout = 5000 dial_timeout = 2500 max_idle_conns_per_host = 100 We highly recommend that you use Categraf as collector for Nightingale.\nFAQ # 1. How to collect multiple instances? # For example, if there are multiple MySQL instances or multiple processes to monitor, how should the configuration be done?\nMost plugin sample configurations in Categraf include a [[instances]] configuration section. For any plugin with this section, you can monitor multiple targets by adding more [[instances]] sections. Categraf\u0026rsquo;s configuration files are in toml format, where double square brackets indicate an array. Take the configuration sample of the MySQL plugin as an example:\n[[instances]] address = \u0026quot;10.1.2.3:3306\u0026quot; username = \u0026quot;categraf\u0026quot; password = \u0026quot;XXXXXXXX\u0026quot; labels = { instance=\u0026quot;n9e-mysql-01\u0026quot; } [[instances]] address = \u0026quot;10.1.2.4:3306\u0026quot; username = \u0026quot;categraf\u0026quot; password = \u0026quot;XXXXXXXX\u0026quot; labels = { instance=\u0026quot;n9e-mysql-02\u0026quot; } Another example is the configuration sample of the process monitoring plugin procstat:\n[[instances]] search_exec_substring = \u0026quot;mysqld\u0026quot; gather_total = true gather_per_pid = true gather_more_metrics = [ \u0026quot;threads\u0026quot;, \u0026quot;fd\u0026quot;, \u0026quot;io\u0026quot;, \u0026quot;uptime\u0026quot;, \u0026quot;cpu\u0026quot;, \u0026quot;mem\u0026quot;, \u0026quot;limit\u0026quot;, ] [[instances]] search_exec_substring = \u0026quot;n9e-plus\u0026quot; gather_total = true gather_per_pid = true gather_more_metrics = [ \u0026quot;threads\u0026quot;, \u0026quot;fd\u0026quot;, \u0026quot;io\u0026quot;, \u0026quot;uptime\u0026quot;, \u0026quot;cpu\u0026quot;, \u0026quot;mem\u0026quot;, \u0026quot;limit\u0026quot;, ] "}),e.add({id:14,href:"/docs/agent/telegraf/",title:"Telegraf",description:"Use Telegraf as collector for Nightingale",content:"Introduction # Telegraf is an agent for collecting, processing, aggregating, and writing metrics. Based on a plugin system to enable developers in the community to easily add support for additional metric collection.\nTelegraf supports multiple output plugins, we can use opentsdb or prometheusremotewrite plugin to send metrics to Nightingale. Below is an example configuration using opentsdb.\nInstall # #!/bin/sh version=1.20.4 tarball=telegraf-${version}_linux_amd64.tar.gz wget https://dl.influxdata.com/telegraf/releases/$tarball tar xzvf $tarball mkdir -p /opt/telegraf cp -far telegraf-${version}/usr/bin/telegraf /opt/telegraf cat \u0026lt;\u0026lt;EOF \u0026gt; /opt/telegraf/telegraf.conf [global_tags] [agent] interval = \u0026quot;10s\u0026quot; round_interval = true metric_batch_size = 1000 metric_buffer_limit = 10000 collection_jitter = \u0026quot;0s\u0026quot; flush_interval = \u0026quot;10s\u0026quot; flush_jitter = \u0026quot;0s\u0026quot; precision = \u0026quot;\u0026quot; hostname = \u0026quot;\u0026quot; omit_hostname = false [[outputs.opentsdb]] host = \u0026quot;http://127.0.0.1\u0026quot; port = 17000 http_batch_size = 50 http_path = \u0026quot;/opentsdb/put\u0026quot; debug = false separator = \u0026quot;_\u0026quot; [[inputs.cpu]] percpu = true totalcpu = true collect_cpu_time = false report_active = true [[inputs.disk]] ignore_fs = [\u0026quot;tmpfs\u0026quot;, \u0026quot;devtmpfs\u0026quot;, \u0026quot;devfs\u0026quot;, \u0026quot;iso9660\u0026quot;, \u0026quot;overlay\u0026quot;, \u0026quot;aufs\u0026quot;, \u0026quot;squashfs\u0026quot;] [[inputs.diskio]] [[inputs.kernel]] [[inputs.mem]] [[inputs.processes]] [[inputs.system]] fielddrop = [\u0026quot;uptime_format\u0026quot;] [[inputs.net]] ignore_protocol_stats = true EOF cat \u0026lt;\u0026lt;EOF \u0026gt; /etc/systemd/system/telegraf.service [Unit] Description=\u0026quot;telegraf\u0026quot; After=network.target [Service] Type=simple ExecStart=/opt/telegraf/telegraf --config telegraf.conf WorkingDirectory=/opt/telegraf SuccessExitStatus=0 LimitNOFILE=65535 StandardOutput=syslog StandardError=syslog SyslogIdentifier=telegraf KillMode=process KillSignal=SIGQUIT TimeoutStopSec=5 Restart=always [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl enable telegraf systemctl restart telegraf systemctl status telegraf "}),e.add({id:15,href:"/docs/agent/datadog-agent/",title:"Datadog-Agent",description:"Use Datadog-Agent as collector for Nightingale",content:"Configuration # Mofidy the configuration item dd_url in the file /etc/datadog-agent/datadog.yaml.\ndd_url: http://nightingale-address/datadog nightingale-address is your Nightingale address.\nRestart # Restart the Datadog-Agent.\nsystemctl restart datadog-agent "}),e.add({id:16,href:"/docs/usage/datasource/",title:"Data Sources",description:"Nightingale monitoring supports integration with various data sources, including Prometheus, ElasticSearch, Grafana Loki, etc. By configuring data sources, Nightingale can query and display monitoring data from these sources, as well as set up alerts based on the data in them.",content:"Nightingale supports integration with various data sources. Early supported data sources, such as Prometheus, VictoriaMetrics, ElasticSearch, etc., support both querying/visualization and alerting. As the project has evolved, Nightingale has positioned itself as an alert engine. Therefore, newly integrated data sources like ClickHouse, MySQL, Postgres, etc., only support alerting and not querying/visualization.\nWhether you want to view data from a data source or set up alerts based on its data, you first need to configure the data source. Add a data source in Integrations - Data sources, select the corresponding data source type, fill in the data source\u0026rsquo;s address, username, password, etc., and click save.\nWhen configuring a data source, in addition to filling in the connection address of the data source, another key point is to select the associated alert engine. If your data source is in an edge computer room and you have built a dedicated n9e-edge for the edge computer room, then select the corresponding n9e-edge as the associated alert engine.\nIn the data source configuration, each form item basically has a tooltip (the small question mark icon next to each form field; you can see usage prompts by hovering the mouse over it), which will not be repeated here.\nAfter configuring the data source, you can check the time series database data on the Ad-hoc Query page. If data can be retrieved, it indicates that the data source configuration is correct.\nFAQ # 1. The writer address of the data source is already configured in Nightingale\u
10026rsquo;s config.toml file. Is it necessary to configure it repeatedly on the page?\nYes. The writer address in config.toml is used for the data forwarding link, while the data source configuration on the page is used for querying and alerting. They are different concepts. In addition, the writer address should be a remote write address, while the data source configuration on the page is usually the base address of the data source. Moreover, many users do not use Nightingale to forward monitoring metrics, so they do not configure the writer address in config.toml and only configure the data source on the page.\n2. I want to use the edge mode to alert on the time series database in the edge computer room, but the central n9e cannot connect to the edge time series database. Can Nightingale still be used for unified alerting in this case?\nYes. Such edge time series databases still need to be added on the page. When adding, select \u0026ldquo;Save\u0026rdquo; instead of \u0026ldquo;Test and Save\u0026rdquo;. This way, the central Nightingale will not verify connectivity and will save successfully directly. At the same time, when configuring the data source, configure the intranet address of the time series database and select an n9e-edge alert engine that can connect to the time series database. Then, n9e-edge will use the intranet address of the time series database for querying and alerting.\nIn this case, the edge time series database can still trigger alerts, but its data cannot be queried on the Nightingale page. Because Nightingale\u0026rsquo;s page queries data through the central n9e, and the central n9e cannot connect to the edge time series database, so queries are not possible.\n"}),e.add({id:17,href:"/docs/usage/ad-hoc/",title:"Ad-hoc Query",description:"Nightingale monitoring supports Ad-hoc queries, allowing direct querying of data source data through the interface. Both time-series metric data and log data can be queried.",content:"Nightingale supports Ad-hoc queries, enabling direct querying of data from data sources through the interface. The menu entry is under Explorer; select Metrics to query metric data and Logs to query log data.\nMetric Query # Here is an example of a metric query:\nThis page is similar to Prometheus\u0026rsquo; graph page, supporting queries for time-series metric data. Of course, some enhancements have been made, such as adding built-in metrics, history records, and other capabilities. In the above image, it is a range vector using the Table view. Nightingale will perform an additional step here to calculate the time difference between each piece of data, which is the +15 on the far right. This helps in troubleshooting data loss issues. For example, if most time differences are regular and consistent with the collection frequency, but suddenly there are two larger time differences that are several times the collection frequency, it indicates a failure in data collection or transmission.\n A common question from new users is why no data is visible when first entering this page. This is expected; you need to enter a PromQL query first to see the data, rather than seeing data immediately upon entering. PromQL is a prerequisite for using Prometheus and Nightingale. It is recommended to learn the basics of PromQL first. For reference materials, see: ãPromql Series Tutorialsã\n If you are using Categraf as the collector, you can query the cpu_usage_active metric. If it can be found, the data source configuration is correct. If you are using Node-Exporter as the collector, you can query the node_load1 metric. If it can be found, the data source configuration is correct.\nLog Query # Log queries mainly support ElasticSearch data sources. When configuring an ElasticSearch data source, many people are confused about the version field. If you are using ElasticSearch 6.x, select 6.0+; if using 7.x, select 7.0+; for higher versions, also select 7.0+. If you encounter compatibility issues, please submit issues for feedback.\nAfter configuring the data source, you can query logs on the Explorer - Logs page. Here is an example of a log query:\nSimilar to Kibana\u0026rsquo;s log query page, Nightingale supports querying by index pattern or directly querying indexes (with wildcard support) without creating an index pattern. However, directly querying indexes is not a good practice and this feature may be removed in the future. Additionally, the query syntax supports both KQL and Lucene (i.e., query string). For ElasticSearch users, these concepts are familiar, so they will not be elaborated here.\nAfter successfully configuring the data source, the next step is to configure alert rules and experience Nightingale\u
10026rsquo;s alert engine capabilities.\n"}),e.add({id:18,href:"/docs/usage/metric-alerting/",title:"Metric Alerting",description:"Nightingale monitoring supports metric alerting. Based on user-configured alert rules, it periodically queries data sources and triggers alerts when the data in the data sources meets the alert thresholds.",content:"Nightingale monitoring divides alerting into two parts: alerting and notification. Alerting refers to the periodic judgment through rules that ultimately generates alert events, while notification refers to the subsequent pipeline and notification process of alert events. This chapter first introduces the alerting part, and we consider it successful when an alert event is finally generated.\nAlerting Principles # Nightingale supports two alerting modes: normal mode and advanced mode (the advanced mode is not open-source yet, but it is planned to be open-sourced later):\n Normal mode: The alert threshold is configured in PromQL, with query conditions and threshold settings comb
1ined. If there are no special requirements, normal mode can be used. This mode has the same alerting logic as Prometheus and has good performance. However, it is a bit troublesome to get the value when the alert is recovered. Advanced mode: Query conditions and threshold settings are separated. If multiple query conditions need to be calculated by addition, subtraction, multiplication, or division, advanced mode can be used. The on-site values of each query condition will be displayed in the alert event, and the value at the time of recovery can also be easily obtained when the alert is recovered. Principles of Normal Mode # In normal mode, Nightingale periodically queries the data source according to the execution frequency configured by the user. The query condition is the PromQL configured by the user, and the query method is an instant query, which calls the /api/v1/query interface of the data source. The number of alert events generated is equal to the number of data points retrieved. For example, if the PromQL is cpu_usage_active \u0026gt; 80, Nightingale uses this PromQL to query the time-series database, and the results returned by the time-series database must be data points where the CPU utilization is greater than 80%, which are all data points that trigger the threshold. Therefore, Nightingale should generate alert events.\nIf the user configures a duration greater than 0 in the alert rule, the process becomes more complex. Nightingale will execute the query multiple times according to the execution frequency within the duration, and generate an alert only if a certain data is found in each query. If the duration is set to 0, it means that an alert will be generated as long as data is found in one query.\nIf an alert event was generated before, but no data is found in subsequent queries, a recovery event will be generated. After all, if no data is found, it indicates that there is no data in the time-series database that meets the threshold condition, so the time-series database no longer returns any data. For alert recovery, there is an advanced configuration called observation duration, which means that after a recovery event is generated, Nightingale will continue to observe for a period of time. If data is found again within the observation duration, no recovery event will be generated (the alert state will continue to be maintained); if no data is found in each query within the observation duration, the recovery event will be finally generated.\nFrom the above analysis, when the alert is recovered, the time-series database does not return any data, so Nightingale cannot get the value at the time of recovery, which is a pain point for many users when using normal mode. Nightingale has designed a way to solve this problem. For details, please refer to the article \u0026ldquo;How to get the value when the alert is recovered?\u0026rdquo;.\nPrinciples of Advanced Mode # In advanced mode, the threshold condition is not put into PromQL. Only filter conditions are written in PromQL. For example, the PromQL:\ncpu_usage_active{cpu=\u0026quot;cpu-total\u0026quot;} In this way, Nightingale uses this PromQL to query the time-series database, and the time-series database returns all data points of CPU utilization each time (with slightly poor performance). Then Nightingale judges the returned data in memory according to the threshold judgment rules configured by the user, as shown in the following figure:\nThe key difference between advanced mode and n
1ormal mode is whether the threshold judgment is done in PromQL and thus by the time-series database, or in Nightingale\u0026rsquo;s memory. In advanced mode, if a recovery event is triggered, the TriggerValue in the recovery event will be automatically filled with the value at the time of recovery. Compared with normal mode, it is easier to obtain the value at the time of recovery.\nIn advanced mode, there will also be a data missing judgment logic, commonly known as NoData alert. Nightingale\u0026rsquo;s behavior is: periodically query the data source, store the found data in memory, and if data is found again in the next query, everything is fine. If a certain piece of data is not found in the next query, an alert will be triggered for that data.\nFunction Description # After understanding the principles, let\u0026rsquo;s configure an alert rule to demonstrate how to use Nightingale\u0026rsquo;s metric alerting function.\nCreation Entry # The menu entry is in Alerts - Rule Management - Alert Rules, as shown in the following figure:\nFirst, select the business group on the left. If there is no business group, you need to create one first. Because there may be many alert rules, they need to be managed by category and subject to permission control, so alert rules are bound to business groups.\n Business groups are a flat list, but can be rendered into a tree structure. As long as the / symbol is used in the business group name, it can be rendered into a tree structure. For example, DBA/MySQL and DBA/Redis will be rendered into the tree style as shown in the above figure. Of course, it is necessary to set the business group display mode to tree and set the business group separator to / in the System Configuration - Site Configuration menu.\n Next, we will focus on explaining the meaning of each configuration item of the alert rule.\n ð¢ Tip: On the rule configuration page, there are tooltips next to each form (small question mark icons, you can see usage prompts when you mouse over them). Please remember to check them.\n Basic Configuration # Rule Name: The name of the alert rule, such as \u0026ldquo;High machine load\u0026rdquo;. Variables can be referenced in the rule name, such as {{ $labels.instance }}, but this is highly not recommended, because it will lead to different names of the finally generated alert events, which is very inconvenient when you want to aggregate and view alert events. Additional Labels: The additional labels configured here will be appended to the labels of the generated alert events, which can be used for aggregation and filtering of alert events later. Remarks: A more detailed description of the alert rule, which supports configuring variables such as $labels and $value. Rule Configuration # Data Source: Select the data source type and filter conditions to specify which data sources the current alert rule takes effect on. Because many companies have multiple sets of Prometheus, this can facilitate rule management. Alert Condition: Configure PromQL. You can perform some condition filtering and four arithmetic operations in PromQL. For example, this PromQL: http_api_request_success{region=\u0026quot;beijing\u0026quot;} / http_api_request_total{region=\u0026quot;beijing\u0026quot;} \u0026lt; 0.995 means: calculate the success rate of all HTTP requests in the beijing region. If the success rate is less than 99.5%, an alert will be triggered. If the alert engine finds data through this PromQL, it indicates that there are abnormal points. If the abnormality persists in multiple queries and finally meets the duration, an alert event will be generated. Multiple Alert Conditions and Level Suppression: In an alert rule, multiple PromQL query conditions can be added. At this time, a level suppression function switch will automatically appear. If the level suppression switch is turned on, if two conditions generate alerts at the same time, only the high-level alert will be sent, and the low-level alert will be suppressed to reduce interference. Execution Frequency and Duration: Both configurations provide tooltips on the page, and you can see usage prompts when you mouse over them. The execution frequency is equivalent to evaluation_interval in Prometheus, and the duration is equivalent to for in Prometheus. If the duration is 0, it means that an alert event will be generated as long as data is found in one query. Event Relabel # The page provides documentation for this part, please refer to it yourself. There is a relabel mechanism in Prometheus, which many people are familiar with (if you haven\u0026rsquo;t learned about it before, you can Google it, it\u0026rsquo;s a very useful design). Prometheus performs relabeling on time-series data, while Nightingale performs relabeling on generated alert events.\nFor example, if there is a label instance=10.1.2.3:9090, you can extract the IP information through relabel to generate a new label ident=10.1.2.3. Nightingale\u
10026rsquo;s alert self-healing function needs to extract machine information from alert events, which is actually the value of the ident label. Extracting machine information through relabel and writing it into the ident label facilitates subsequent alert self-healing (provided that you configure hostname=\u0026quot;$ip\u0026quot; in Categraf).\nEffective Configuration # This part also provides usage instructions on the page, please refer to it yourself. The most important configuration here is the effective time period. For example, an alert rule can only take effect during the day, and another alert rule can only take effect at night, which can be configured through the effective time period.\nNotification Configuration # In the old version, alert recipients and notification media were directly configured in the alert rules, which was cumbersome to modify in batches. The new version extracts the notification logic and abstracts it into notification rules to handle all logic after an alert event is generated. The notification rules will be introduced in detail later.\nEach other field in the notification configuration has a tooltip, and you can see usage prompts when you mouse over them, so they will not be repeated here.\nAlert self-healing: After an alert is generated, automatically execute a specific script on the alerted machine (or a specified central control machine). Where does the information of the alerted machine come from? It is taken from the ident label of the alert event. Which self-healing script to execute specifically? It is specified through the Alert Self-healing field under the notification configuration.\nAdditional information is similar to Annotations in Prometheus alert rules. After an alert event is generated, Nightingale will append this additional information to the alert event, which can be referenced in the message template later and finally displayed in notifications such as DingTalk, Feishu, email, etc.\nPractical Demonstration # In order to generate an alert event as soon as possible, I configured a PromQL that will definitely trigger:\ncpu_usage_active \u0026gt; 0 ð¢ The indicator cpu_usage_active is collected by Categraf, which represents CPU utilization. Obviously, CPU utilization will definitely be greater than 0, so this rule can trigger an alert event soon. If you are not using Categraf, you will not have this indicator. Please use the indicators in your own time-series database for testing.\n In the above example, to speed up the generation of alert events, I set the execution frequency to 15s and the duration to 0. In this way, Nightingale will query the data source every 15s, and generate an alert event if data is found.\nAfter a while, you can find that the status field on the left side of the alert rule becomes a red exclamation mark, indicating that an alert event has been triggered. Clicking on it will show the relevant alert events generated by this rule in the side panel. Of course, you can also see the current active alerts (unrecovered alerts are called active alerts) and all historical alerts in the alert events menu.\nThe first alert event above is the one just tested, and the other events are from previous tests, so you don\u0026rsquo;t need to care about them. The alert event is generated, indicating that the alert rule is configured correctly. Next, configure the notification rule to specify who will receive what kind of alert events and through which notification media (phone calls, SMS, emails, Feishu, DingTalk, WeChat Work, etc., called notification media).\n"}),e.add({id:19,href:"/docs/usage/logs-alerting/",title:"Log Alerting",description:"Nightingale monitoring supports log alerting, which allows configuring alert rules for log data in ElasticSearch, Loki, and ClickHouse. It periodically queries the data sources and triggers alerts when the data in the data sources meets the alert thresholds.",content:"Nightingale monitoring supports log alerting, which allows configuring alert rules for log data in data sources such as ElasticSearch, Loki, and ClickHouse. It periodically queries the data sources and triggers alerts when the data in the data sources meets the alert thresholds. The main difference between log alerting and metric alerting lies in the writing of query conditions. Other fields are common, so please be sure to read the content of Metric Alerting first, which will not be repeated here.\nElasticSearch Alerting Principle # ElasticSearch supports different query syntaxes, such as DSL, KQL, Lucene, EQL, SQL, etc. As an alert engine, Nightingale essentially allows users to configure query statements, then periodically queries the data sources, makes threshold judgments on the found data, and triggers alerts when the threshold conditions are met.\nThe first supported query syntax is Lucene, i.e., the query_string method. After finding the data, a basic statistics is performed, such as counting the number of log lines found, or making statistics on a certain field in the logs to calculate its average, maximum value, quantile, etc. Then the statistical results are compared with the thresholds configured by the user, and an alert is triggered when the threshold conditions are met.\nElasticSearch Alert Configuration # First, configure the data sources, that is, which ElasticSearch data sources the current alert rule will take effect on. This is essentially the same logic as metric alerting.\n ð¢ Only the data sources configured with the corresponding type will be displayed here in the data source type. That is: if you only configure Prometheus-type data sources, you will not see other types such as ElasticSearch, TDEngine, ClickHouse when creating alert rules.\n Then configure the query and statistical conditions:\nFirst, select the index, which supports wildcards. I configured fc* above. Then the most critical part is the filter condition. I configured message.status:\u0026gt;100 above to filter logs where the message.status field is greater than 100. This filter condition uses Lucene syntax, which is different from Kibana\u0026rsquo;
1s query syntax. Logs must have a date field, and you need to tell Nightingale which field is the date field through configuration, then configure a time interval. In the above figure, it is configured as 5 minutes, and Nightingale will filter the data of the last 5 minutes according to the date field.\nThen there is the statistical analysis method. In the above example, count is selected, which means counting the number of log lines without any Group By conditions.\nFinally, there is the threshold judgment, that is, comparing the result of the above count with the threshold, and generating an alert event if the condition is met. In the above example, the threshold is \u0026gt; 0 to trigger an alert, that is, if the result of count is greater than 0, an alert will be triggered.\nAfter a short while, we can see the generated alert events:\nElasticSearch Filter Conditions # What other ways can the filter conditions (i.e., message.status:\u0026gt;100 in the above example) be written? You can click the small question mark icon next to the filter condition, and a side panel will display sample explanations of the filter condition writing methods. Some typical writing methods are as follows:\n status:active Query records where the status field contains \u0026ldquo;active\u0026rdquo; title:(quick OR brown) Query records where the title field contains \u0026ldquo;quick\u0026rdquo; or \u0026ldquo;brown\u0026rdquo; author:\u0026quot;John Smith\u0026quot; Query records where the author field contains the exact phrase \u0026ldquo;John Smith\u0026rdquo; count:[1 TO 5] Range query, closed interval, i.e., including 1 and 5 date:[2022-01-01 TO 2022-12-31] Date range query age:\u0026gt;=10 Numeric filtering, greater than or equal to 10 Note that to avoid mistakes, it is recommended not to add spaces before or after the colon : following the field. In addition, different conditions can be connected using AND or OR, such as status:active AND age:\u0026gt;=10. For more writing methods, please refer to the official ElasticSearch documentation.\n"}),e.add({id:20,href:"/docs/usage/notify-rules/",title:"Notification Rules",description:"Nightingale monitoring supports notification rules, which allow configuration of notification methods, recipients, etc. When an alarm event is triggered, Nightingale will send notifications according to the notification rules. For example, high-level alarms can be sent via phone calls, SMS, or DingTalk, while low-level alarms can be sent via email.",content:"Alarm rules are responsible for generating alarm events, and notification rules are responsible for sending out alarms. Different alarms can use different notification media. For example, high-level alarms can be sent via phone calls, SMS, or DingTalk, while low-level alarms can be sent via email.\nDesign Purpose # Older versions of Nightingale did not have the concept of notification rules. Notification media and recipients were directly configured in alarm rules. Although this was intuitive, it lacked flexibility, with the following issues:\n When suppression was enabled in alarm rules, the notification media were still hard-coded. In previous versions, enabling suppression rules usually meant that different thresholds required different levels, and thus different notification media. For example, Critical-level alarms might use phone calls or SMS, while Info-level alarms used Email. However, in previous versions, the notification media were fixed and could not vary by level. Integrating notification methods like phone calls and SMS was inconvenient. This version provides universal HTTP and script-based sending methods, with customizable HTTP parameters, headers, and bodies, making it easier to integrate different notification media. Previous notification methods were tightly coupled with alarm rules, making modifications cumbersome. The new version introduces the concept of \u0026ldquo;notification rules.\u0026rdquo; Alarm rules are directly associated with notification rules, and the specific sending methods are defined in the notification rules. This decouples alarm rules from notification rules. Multiple alarm rules can be associated with a single notification rule, so changing the notification method only requires modifying the notification rule, affecting all associated alarm rules. Message templates in previous versions were rigid, with each type of notification medium limited to a fixed template. The new version supports custom message templates, and each notification medium can be associated with different templates. For example, the DBA team and Big Data team can both use DingTalk robots for alarms but with different message templates. Logic Diagram # The overall process for sending alarm events in the new version is as follows:\nIn previous versions, notification media and recipients were directly configured in alarm rules, leading to tight coupling. In the new version, alarm rules are associated with notification rules, and the sending details are defined in the notification rules. This decoupling allows multiple alarm rules to reference a single notification rule, simplifying updates to notification methods.\nConfiguration Instru
1ctions # Notification rules support different notification media and can define the scope of application for each medium. For example, phone calls may only apply to Critical alarms, while Email may apply to Critical, Warning, and Info alarms. Below is a sample configuration of a notification rule:\nWe provide some built-in notification media for out-of-the-box use:\nWhen configuring a notification medium, the \u0026ldquo;Variable Configuration\u0026rdquo; may be confusing. Consider this scenario: Both the DBA team and BigData team want to use WeCom for alarms but with different WeCom robots (i.e., similar Webhook URLs but different Key parameters in the URL, where each Key represents a different robot). How to handle this?\nIn Nightingale\u0026rsquo;s design, we avoid creating two separate notification media. Instead, we use a single WeCom medium that supports parameters. DBA team members configure the WeCom medium in their notification rules and provide their robot\u0026rsquo;s Key, while BigData team members do the same with their Key. This way, one medium supports multiple robots.\nTo enable parameter support for a notification medium, configure variables in the medium\u0026rsquo;s settings. The built-in WeCom medium, for example, has two parameters: Key (WeCom robot Key) and Bot Name (a custom name for easy reference, similar to a note). These parameters can then be referenced in the medium\u0026rsquo;s HTTP configuration:\nThis scenario is straightforward: the medium retrieves the Key provided by the user. For more complex scenarios, such as sending SMS, how should medium parameters be defined? If we directly define a Phone parameter and require users to enter phone numbers manually in notification rules, it would be cumbersome. Additionally, if a user\u0026rsquo;s contact information changes, they would need to update both their profile and the notification rule, which is inefficient. Since phone numbers are already stored in user profiles, we can link the two.\nFor such scenarios, medium parameters can be derived from user Profile information. This requires referencing user Profile data in the medium\u0026rsquo;s variable configuration. For example:\nIn the medium variables, selecting \u0026ldquo;Phone\u0026rdquo; as the contact method enables the following functionality:\n In the notification rule, the system recognizes that the medium requires phone numbers from user profiles, allowing selection of contacts or teams instead of manual entry of phone numbers. In the HTTP request body or query string, the magic variable {{ $sendto }} can be referenced, representing the recipient\u0026rsquo;s phone number. This allows the medium to send alarms to the correct recipient. The {{ $sendto }} design is inspired by Zabbix, so users familiar with Zabbix will find it intuitive. Configuration Examples # Integrating DingTalk Alerts Integrating WeCom Alerts Integrating Feishu Alerts Integrating DingTalk Alerts: How to Configure Mentions Integrating DingTalk, Feishu, and WeCom Notifications Integrating Alibaba Cloud SMS Additionally, a video tutorial on integrating Feishu alerts is available on the WeChat Channels: SRETALK. You can search for it to watch.\nEmail Alarm Configuration # The links above cover common IM notification media. Here, weè¡¥å
(add) instructions for configuring email alarms. To send alarm emails, three key configurations are required:\n SMTP server settings Recipients\u0026rsquo; email addresses in their profiles Email notification message templates 1. SMTP Server Configuration # Navigate to the Media types menu, find the built-in Email medium, and edit it to enter SMTP server details. An example configuration is shown below:\n2. Users Configure Their Email Addresses in Personal Profiles # Click the avatar in the top-right corner to access the personal information page and enter the email address for receiving alarms.\n3. Configure Notification Rules # For testing, create a new notification rule (or modify an existing one to add the email medium) that applies to all alarm levels, ensuring every alarm triggers an email notification. Example configuration:\nThe example above sends notifications to an individual, but you can also select a team. Click \u0026ldquo;Test Notification\u0026rdquo; and choose a historical alarm event to test email delivery. Finally, associate this notification rule with your alarm rules.\n4. Email Notification Message Template # In the Message templates menu, find the Email template, which includes two variables:\n content: Email body content subject: Email subject Both fields use Go template syntax, allowing customization of email content and subject as needed.\n"}),e.add({id:21,href:"/docs/usage/dashboard/",title:"Dashboard",description:"Nightingale monitoring supports dashboard functionality, which can display monitoring data in the form of charts. Through the dashboard, users can intuitively view the changing trends and status of various monitoring metrics.",content:"Although Nightingale monitoring focuses on alerting, it also supports dashboard functionality. While it may not be as sophisticated as Grafana, it supports common chart types and can meet daily usage needs.\nQuick Import # We have previously organized machine dashboards that you can directly import to quickly see the effects. The import method:\nDifferent c
1ollectors collect monitoring metrics with different names and labels, so separate dashboards need to be created for each. If you are using Categraf, you can import the following two dashboards:\n Machine overview data: categraf-overview.json Machine detailed data: categraf-detail.json If you are using Node Exporter, you can import the following dashboard:\n Node key metrics: exporter-detail.json In fact, these dashboards can be found in the Nightingale menu Integration Center - Template Center - Search for Linux:\nSample of Categraf overview dashboard:\nSample of Categraf machine detailed data dashboard:\nThe template center already has built-in dashboards for many components, but their quality varies. We will reorganize them one by one when we have time, striving for out-of-the-box usability. However, there are too many components and limited manpower. We welcome community users to participate in organizing and contributing. You can submit your organized dashboards to the dashboards directory under each component in the integrations directory of the Nightingale Github repository via PR.\nIntegrating Grafana # You can also continue using Grafana for visualization, as each has its own strengths, and using them in combination works better. You can also embed Grafana into Nightingale via iframe through the Nightingale menu Integration Center - System Integration.\nThere are two embedding methods: one is secure embedding through SSO for user login, and the other is embedding via anonymous access.\nSecure Embedding # Please refer to this article: Deep Integration of Nightingale and Grafana with Authentication\nAnonymous Embedding # Grafana does not support being embedded by other systems by default. Some configurations need to be modified as follows:\n Enable embedding Find the allow_embedding configuration item in the Grafana configuration file and set it to true.\nEnable anonymous access Find the auth.anonymous configuration section, set enabled to true, org_role to Viewer, and configure org_name according to your own environment.\nFor HTTPS-enabled Grafana In the security configuration section, set cookie_secure to true and cookie_samesite to none.\nIn addition, the configuration methods may vary between different versions of Grafana. If you find any errors in the documentation, you can click \u0026ldquo;Edit this page on GitHub\u0026rdquo; below to edit this page directly and submit a PR to modify the documentation.\n"}),e.add({id:22,href:"/docs/practice/linux/",title:"Linux OS",description:"Nightingale monitoring supports monitoring of Linux hosts. It can collect various indicator data of hosts through collectors such as Categraf or Node Exporter, display them in dashboards, and use Nightingale's alerting capabilities for alert configuration.",content:" For monitoring systems, the strength of basic functions is indeed crucial, but how to implement them in different scenarios is even more critical. In the \u0026ldquo;Monitoring Practice\u0026rdquo; chapter, we collect various monitoring practice experiences, which will be categorized by different components. If you have good practice experiences with a certain component, you are welcome to submit a PR and attach your article link to the corresponding component directory.\n Best Practices for Linux Host Monitoring Thoroughly Understand Machine Monitoring Through Node-Exporter FAQ # 1. I can see the machines in my machine list and also see information like CPU and memory of the machines, but no data can be found in the dashboard # ð¡ Note: The CPU, memory and other information in the machine list are not stored in the time-series database, but in Redis. They are reported when Categraf calls Nightingale\u0026rsquo;s heartbeat interface, which is a different path from Remote write.\n Troubleshoot this issue from the following aspects:\n Check Categraf\u0026rsquo;s logs As IT practitioners, the first reaction should be to check the logs of related components. Categraf\u0026rsquo;s logs are output to stdout by default. If Categraf is managed by systemd, use journalctl to view them, such as journalctl -u categraf.service. If you are not very familiar with Linux, you can directly start Categraf in the foreground on the command line to view the logs more conveniently:\n./categraf Starting the Categraf process in the foreground as above will output the logs directly to the terminal for easy viewing.\nConfirm Categraf\u0026rsquo;s configuration The fact that you can normally see content in the machine list indicates that the heartbeat configuration in Categraf\u0026rsquo;s configuration is normal. If no monitoring data is visible in the dashboard, there may be a problem with the writer configuration. The url in the writer part should be configured as the address of Nightingale, and the urlpath is /prometheus/v1/write.\nConfirm Nightingale\u
10026rsquo;s configuration Categraf pushes data to Nightingale, and Nightingale does not store data directly but forwards it to TSDB, which can be Prometheus or VictoriaMetrics, etc. Which TSDBs does Nightingale send data to? It is determined by Pushgw.Writers in Nightingale\u0026rsquo;s configuration file config.toml.\nIt is necessary to ensure that the configuration in Pushgw.Writers is correct and that Nightingale\u0026rsquo;s n9e process can normally access these TSDBs.\nCheck Nightingale\u0026rsquo;s logs If data forwarding to the time-series database fails, Nightingale\u0026rsquo;s logs will have relevant prompts. Checking Nightingale\u0026rsquo;s logs can help locate the problem. A common mistake among new users in the community is that Nightingale writes data to Prometheus, but Prometheus has incorrect startup parameters and does not enable the remote write interface, resulting in Nightingale\u0026rsquo;s failure to write data. Such errors are usually prompted in Nightingale\u0026rsquo;s logs, and you can directly see what parameters should be added to Prometheus, just modify them accordingly.\nTime synchronization For example, check whether the time of the local laptop is consistent with that of the server. Monitoring systems are very sensitive to time. If the time is not synchronized, it may cause data to not be displayed normally.\nCheck the dashboard configuration Some dashboards view all data in the time-series database, while others can only view monitoring data of machines under the business group (controlled by dashboard variables). For the latter type of dashboard, it is necessary to ensure that there are machines under the business group.\n2. Can I write monitoring data to other time-series databases such as TDEngine? # First, you need to understand the Prometheus remote write protocol (you can ask Google or GPT). The data collected by Categraf is pushed to Nightingale through the Prometheus remote write protocol, and Nightingale also forwards data to the time-series database through the Prometheus remote write protocol.\nTherefore, if a time-series database supports receiving data via the Prometheus remote write protocol, it can be connected to Categraf or Nightingale. Where to get this information? Check (or search) the time-series database\u0026rsquo;s documentation. If it supports receiving data via the Prometheus remote write protocol, it will most likely be mentioned in the documentation. If it is not written in its documentation, it is probably not supported or not well-supported and not recommended.\n3. How to monitor machine disconnection? # In Prometheus, each machine deploys Node-Exporter, and Prometheus actively fetches data from Node-Exporter. This method is called PULL. The advantage of this method is that Prometheus can know whether the machine is disconnected, because if the machine is disconnected, Prometheus cannot fetch data. If the fetch is successful, there will be an up indicator with a value of 1; if the fetch fails, the value of the up indicator is 0.\nTherefore, in Prometheus PULL mode, the up indicator can be used to monitor whether the machine is disconnected.\nNightingale uses Categraf to collect machine monitoring data by default. Categraf does not expose the /metrics interface, but pushes data to Nightingale through the remote write protocol, which is called PUSH mode. In this mode, there will be no up indicator, so how to monitor machine disconnection?\nNightingale\u0026rsquo;s alert rules provide a Host type alert rule, which can be configured for disconnection alerts:\nIt is usually configured to take effect for all machines. If you have some special machines for which you do not want to set disconnection alerts, you can put these machines into a special business group or mark them with special tags, and then filter them out in the machine selection here.\nAlternatively, use PING monitoring to initiate PING detection on the machine and configure alert rules for the PING detection results. Many monitoring tools support PING detection, such as Telegraf, Categraf, Blackbox Exporter, etc.\n"}),e.add({id:23,href:"/docs/practice/proc/",title:"Process",description:"This article introduces practical experience in process monitoring, including overall process count statistics and single process metric collection. It provides detailed explanations of Categraf's processes plugin and procstat plugin, along with links to related dashboards.",content:" For monitoring systems, the strength of basic functions is indeed crucial, but how to implement them in different scenarios is even more critical. In the \u0026ldquo;Monitoring Practice\u0026rdquo; chapter, we collect various monitoring practices, categorized by different c
1omponents. If you have good practical experience with a certain component, please submit a PR and attach your article link to the corresponding component directory.\n Process monitoring consists of two parts: one is the statistics of the total number of processes in the operating system, and the other is the collection of metrics for individual processes.\nOverall Process Count # Taking Categraf as an example, Categraf provides the processes plugin for counting the number of processes on a machine, such as the total number of processes, the number of processes in Running state, the number of processes in Sleeping state, etc. For the data collected by the processes plugin, we have sorted out a dedicated dashboard:\n https://github.com/ccfos/nightingale/blob/main/integrations/Linux/dashboards/categraf-processes.json\n What is the use of such metrics? They are usually useful in scenarios where a large number of unexpected processes are started. For example, the author once encountered a situation where a crontab script was poorly written and hung, and there was no check in the cron script to see if the previous process had exited. As a result, each time crontab executed, a new process was started, eventually leading to a large number of processes with the same name running on the machine, which eventually caused an accident. At this time, the metrics collected by the processes plugin can be used to detect the problem.\nSingle Process Metrics # Single process metrics refer to indicators such as CPU, memory, and file handles occupied by a process. There are multiple ways to collect them.\n Embedding monitoring points in the process. For example, Java programs can use micrometer or Spring Boot Actuator to collect metrics, and Go programs can use Prometheus\u0026rsquo;s Go client library to collect metrics. Collecting outside the process. For example, using Process Exporter, Categraf\u0026rsquo;s procstat plugin, etc., to collect process metrics. In general, embedding monitoring points in the process is a more recommended approach. It can not only collect conventional metrics such as CPU and memory of the process but also collect more runtime metrics. For example, Java programs can collect some JVM metrics, and Go programs can collect some goroutine and gc metrics. All excellent open-source software will expose their own monitoring metrics. As business developers have varying levels of proficiency, some may not be aware of the importance of embedding monitoring points. In such cases, out-of-process collection can be used as a supplement.\n Spring Boot Actuator can be configured to directly expose metrics data in Prometheus format, so no additional plugins are needed for c
1ollection. You can directly use Categraf\u0026rsquo;s prometheus plugin. Alternatively, you can configure scraping rules directly in Prometheus or vmagent.\n Taking Categraf\u0026rsquo;s procstat plugin as an example, its documentation can be referenced here. The key metrics to focus on are:\n procstat_lookup_count: the number of processes. If it is 0, it means the corresponding process has crashed. procstat_rlimit_num_fds_soft: the soft limit of file handles for the process. If it is 1024, it usually indicates that the system parameters are not properly tuned. procstat_cpu_usage_total: process CPU usage. procstat_mem_usage_total: process memory usage. procstat_num_fds_total: the total number of file handles opened by the process. procstat_read_bytes_total: the total number of bytes read by the process. procstat_write_bytes_total: the total number of bytes written by the process. For a dashboard for single processes, you can refer to:\n https://github.com/ccfos/nightingale/blob/main/integrations/Procstat/dashboards/categraf-procstat.json\n FAQ # 1. How to monitor multiple processes with the procstat plugin?\nA configuration example is as follows:\n[[instances]] search_exec_substring = \u0026quot;mysqld\u0026quot; gather_total = true gather_per_pid = true gather_more_metrics = [ \u0026quot;threads\u0026quot;, \u0026quot;fd\u0026quot;, \u0026quot;io\u0026quot;, \u0026quot;uptime\u0026quot;, \u0026quot;cpu\u0026quot;, \u0026quot;mem\u0026quot;, \u0026quot;limit\u0026quot;, ] [[instances]] search_exec_substring = \u0026quot;n9e-plus\u0026quot; gather_total = true gather_per_pid = true gather_more_metrics = [ \u0026quot;threads\u0026quot;, \u0026quot;fd\u0026quot;, \u0026quot;io\u0026quot;, \u0026quot;uptime\u0026quot;, \u0026quot;cpu\u0026quot;, \u0026quot;mem\u0026quot;, \u0026quot;limit\u0026quot;, ] 2. What is the purpose of the jvm parameter in gather_more_metrics in the procstat configuration?\nIf gather_more_metrics includes jvm, it will be considered that the target process to be collected is a Java process, and the system\u0026rsquo;s jstat command will be called to collect some basic JVM metrics. jstat is a tool that comes with the JDK installation, located in the bin directory of the JDK. A common pitfall here is that users configure jvm in gather_more_metrics, have jstat on the machine, and can collect data when testing with the following command:\n./categraf --test --inputs procstat But after restarting Categraf for formal collection, data cannot be collected. The usual reason is that Categraf is managed by systemd, and systemd does not know the JDK environment variables, so it cannot find the jstat command. The solution is to configure Categraf\u0026rsquo;s service file and add the JDK environment variables. For example:\nEnvironment=\u0026quot;PATH=/usr/lib/jvm/java-11-openjdk-amd64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\u0026quot; "}),e.add({id:24,href:"/docs/practice/port/",title:"Port",description:"This article introduces practical experience in port monitoring, including port monitoring for TCP/UDP protocols and HTTP protocol. Use Categraf's net_response and http_response plugins to implement port monitoring, and provides links to related dashboards.",content:" For monitoring systems, the strength of basic functions is indeed crucial, but how to implement them in different scenarios is even more critical. In the \u0026ldquo;Monitoring Practice\u0026rdquo; chapter, we collect various monitoring practice experiences, which will be categorized by different components. If you have good practical experience with a certain component, please submit a PR and attach your article link to the corresponding component directory.\n Port monitoring is a typical method for process liveness detection. Compared with counting the number of processes, port monitoring is more reliable because processes may sometimes hang, resulting in normal process count statistics but unresponsive ports.\nGenerally speaking, port detection is divided into three protocols:\n TCP protocol UDP protocol HTTP protocol Depending on the port protocol type that the service listens to, different detection methods are used.\nTCP/UDP Protocols # Port monitoring for TCP/UDP protocols is suitable for RPC services and can be implemented using Categraf\u0026rsquo;s net_response plugin.\n Categraf net_response Plugin Description The most important metric to focus on here is: net_response_result_co
1de. If the value of this metric is 0, it indicates everything is normal; if it is non-zero, it indicates an exception, and different values represent different types of exceptions.\n 0: Success 1: Timeout 2: ConnectionFailed 3: ReadFailed 4: StringMismatch Relevant dashboards can be found in Nightingale\u0026rsquo;s Integrations - Components.\nHTTP Protocol # The detection of the HTTP protocol is similar to that of the TCP/UDP protocols, and Categraf also provides the http_response plugin for implementation. Compared with TCP/UDP protocols, port monitoring for the HTTP protocol can go a step further. In addition to detecting whether the port is available, it can also detect whether the HTTP response content (returned status code, returned Response body) meets expectations. For HTTPS sites, it can also detect the certificate expiration time.\n Categraf http_response Plugin Description The metric used for alarming is http_response_result_code. As long as this metric is 0, it is normal; if it is non-zero, it is abnormal, and different values represent different meanings:\nSuccess = 0 ConnectionFailed = 1 Timeout = 2 DNSError = 3 AddressError = 4 BodyMismatch = 5 CodeMismatch = 6 http_response_cert_expire_timestamp is the timestamp of the certificate expiration time. http_response_cert_expire_timestamp - time() indicates how long it will be until the certificate expires, in seconds.\nRelevant dashboards can be found in Nightingale\u0026rsquo;s Integrations - Components.\n"}),e.add({id:25,href:"/docs/practice/exec/",title:"Plugin Scripts",description:"This article introduces the mechanism of custom monitoring plugin scripts in Categraf. Plugin scripts can be written in Shell, Python, Go, etc. Simply tell Categraf the path of the executable file, and Categraf will periodically execute these scripts at the configured time interval and collect the output as monitoring data.",content:" For a monitoring system, the strength of basic functions is indeed crucial, but how to implement them in different scenarios is even more critical. In the \u0026ldquo;Monitoring Practice\u0026rdquo; chapter, we collect various monitoring practice experiences, which will be categorized by different components. If you have good practical experience with a certain component, you are welcome to submit a PR and attach the link of your article to the corresponding component directory.\n Although Categraf has many built-in collection plugins, there are always some scenarios where custom monitoring data collection is required. In such cases, you can consider using Categraf\u0026rsquo;s input.exec plugin. This plugin can execute user-specified scripts (which can be Shell, Python, Perl scripts, or binary files of Go, C++, as long as they are executable files), then capture the stdout of the script and parse it into monitoring data.\n EXEC Plugin User Documentation Some community users have previously provided some plugin script examples for reference: Categraf Exec Plugin Script Examples. Everyone is welcome to continue submitting examples.\n"}),e.add({id:26,href:"/docs/practice/mysql/",title:"MySQL",description:"This article introduces how to monitor MySQL. As one of the most commonly used databases in China, MySQL is widely used and has a lot of related materials. Monitoring MySQL involves two aspects: one is monitoring MySQL's performance data, and the other is monitoring MySQL's business data.",content:" For monitoring systems, the strength of basic functions is indeed crucial, but how to implement them in different scenarios is even more critical. In the \u0026ldquo;Monitoring Practice\u0026rdquo; chapter, we collect various monitoring practice experiences, which will be categorized by different components. If you have good practical experience with a certain component, you are welcome to submit a PR and attach the link of your article to the corresponding component directory.\n Getting Started Tutorial for Monitoring MySQL with Categraf Building MySQL Monitoring According to the Guidance of Nightingale Template Center How to Detect and Handle MySQL Master-Slave Delay Issues Explanation of MySQL Monitoring Principles Monitoring MySQL with Exporter Solving the max_prepared_stmt_count Problem in MySQL In addition to monitoring MySQL\u0026rsquo;s performance data, we can also customize SQL to monitor data in MySQL (which can be done through Categraf\u0026rsquo;s mysql plugin). This usually has two purposes:\n Extending performance monitoring metrics. If the default performance monitoring data is insufficient, this method can be used for extension. Monitoring business data. This scenario is extremely extensive, such as monitoring order data, user data, etc. This scenario is easily overlooked but can sometimes have surprising effects. "}),e.add({id:27,href:"/docs/practice/redis/",title:"Redis",description:"There are various methods to collect Redis monitoring data, such as using tools like Categraf, Redis-Exporter, and Cprobe. The principle is similar: connecting to the Redis instance and executing commands like `info` to obtain monitoring data.",content:" For a monitoring system, the strength of basic functions is indeed crucial, but how to implement them in different scenarios is even more important. In the \u0026ldquo;Monitoring Practice\u0026rdquo; chapter, we collect various monitoring practice experiences, categorized by different c
1omponents. If you have good practical experience with a certain component, please submit a PR and attach the link to your article in the corresponding component directory.\n Collect Redis Monitoring Data with Categraf Monitor MySQL, Redis, MongoDB, Oracle, Postgres, etc. with Cprobe Principle of Redis Monitoring Data Collection # Whether using Categraf or Redis-Exporter to collect Redis monitoring data, the principle is similar: connect to Redis using information such as the Redis connection address, username, and password, and execute commands like info to obtain monitoring data.\nHow to Connect Redis-Exporter # Some users use Categraf to collect machine metrics, process metrics, and custom plugins, but do not use Categraf to collect Redis monitoring data; instead, they use Redis-Exporter. Then they are confused: how to connect the data collected by Redis-Exporter to Nightingale?\nThere are two methods:\n Configure Scrape rules directly in your time-series database to scrape data from Redis-Exporter Use the input.prometheus plugin of Categraf to scrape data from Redis-Exporter "}),e.add({id:28,href:"/docs/practice/oracle/",title:"Oracle",description:"There are various methods to collect Oracle monitoring data, such as using tools like Categraf and Cprobe. The underlying principle is similar: connecting to an Oracle instance and executing relevant commands to obtain monitoring data.",content:" For monitoring systems, the strength of basic functions is indeed crucial, but how to implement them in different scenarios is even more important. In the \u0026ldquo;Monitoring Practice\u0026rdquo; chapter, we collect various monitoring practice experiences, categorized by different components. If you have valuable practice experiences with a specific component, please submit a PR and attach a link to your article in the corresponding component directory.\n There are various methods to collect Oracle monitoring data, such as using tools like Categraf and Cprobe. The underlying principle is similar: connecting to an Oracle instance and executing relevant commands to obtain monitoring data. This article uses Categraf v0.4.15 or later as an example to introduce the configuration method for Oracle monitoring data collection.\nOverview of Oracle Plugin Configuration # All plugin configurations for Categraf are by default in the conf directory. The configuration directory for Oracle is conf/input.oracle, which contains two configuration files:\n oracle.toml: The main configuration file for the Oracle plugin, which configures connection and authentication information for different Oracle instances. Categraf can connect to multiple Oracle instances simultaneously, and the configuration file can include connection information for multiple instances through different [[instances]] configuration sections. metric.toml: The Oracle plugin collects Oracle monitoring data by executing various SQL statements. Some SQL statements are general and intended to be executed by all Oracle instances, while others are specific to certain instances. General SQL statements are configured in metric.toml, and instance-specific SQL statements are configured in oracle.toml. oracle.toml # A sample configuration for oracle.toml is as follows:\n# Default collection frequency; all configured Oracle instances will use this frequency by default # If an instance requires a different collection frequency, adjust it using interval_times in the instance configuration # The final collection frequency for each instance = interval * interval_times # If interval is not configured here, the global interval in Categraf's configuration will be used (default is 15 seconds) # Unit is seconds, so the default is to collect monitoring data every 15 seconds interval = 15 # Configuration for the first Oracle instance, enclosed in [[instances]] # [[instances]] uses double brackets, which in TOML represent an array # Multiple [[instances]] blocks can be configured, meaning multiple Oracle instances can be set up [[instances]] address = \u0026quot;10.1.2.3:1521/orcl\u0026quot; username = \u0026quot;monitor\u0026quot; password = \u0026quot;123456\u0026quot; is_sys_dba = false is_sys_oper = false disable_connection_pool = false max_open_connections = 5 # The final collection frequency for this instance is interval * interval_times interval_times = 1 # Additional dimension labels can be attached to this instance, which will be added to the monitoring data of the current instance labels = { region=\u0026quot;cloud\u0026quot; } # The metrics configuration section under instances specifies the monitoring data to be collected for the current instance # Note that this metrics configuration section is specific to the current instance; other instances will not execute these SQL statements [[instances.metrics]] mesurement = \u0026quot;sessions\u0026quot; label_fields = [ \u0026quot;status\u0026quot;, \u0026quot;type\u0026quot; ] metric_fields = [ \u0026quot;value\u0026quot; ] timeout = \u0026quot;3s\u0026quot; request = ''' SELECT status, type, COUNT(*) as value FROM v$session GROUP BY status, type ''' [[instances]] address = \u0026quot;192.168.10.10:1521/orcl\u0026quot; username = \u0026quot;monitor\u0026quot; password = \u0026quot;123456\u0026quot; is_sys_dba = false is_sys_oper = false disable_connection_pool = false max_open_connections = 5 labels = { region=\u0026quot;local\u0026quot; } # The second instance has no corresponding instances.metrics configuration section, indicating no instance-specific collection SQL # That is: the second instance will only execute the general SQL configured in metric.toml Principle of Oracle monitoring data collection: Periodically execute SQL, convert the returned results into Prometheus time-series data format, and send them to the server. The result of SQL execution is a two-dimensional table with multiple rows and columns. We need to configure Categraf to specify which columns serve as labels for the time-series data and which serve as metric values.\
1n mesurement: A custom metric prefix request: The SQL statement for querying monitoring data label_fields: Columns from the SQL result that will be used as labels for the time-series data metric_fields: Columns from the SQL result that will be used as metric values field_to_append: Whether to append the content of a column to the monitoring metric name as a suffix timeout: Timeout duration for SQL execution ignore_zero_result: Whether to ignore rows with a value of 0 in the query results. If not ignored (set to false) and no data is found, an error log will be printed. If ignored (set to true), no error log will be printed when no data is found. metric.toml # This file configures some commonly used SQL statements for Oracle monitoring data collection. Categraf will execute these SQL statements regularly to obtain monitoring data from all Oracle instances. The specific meanings and functions of these SQL statements are likely more familiar to Oracle DBAs. Oracle DBAs are welcome to write articles explaining these SQL statements and their uses. You can submit a PR with a link to your article on this page to benefit more people.\n"}),e.add({id:29,href:"/docs/practice/java/",title:"Java Programs",description:"How to monitor Java programs? There are usually the following methods: Actuator, Micrometer, JMX, and components exposing metrics themselves. This article introduces the configuration methods and usage scenarios of these methods.",content:" For monitoring systems, the strength of basic functions is indeed crucial, but how to implement them in different scenarios is even more critical. In the \u0026ldquo;Monitoring Practice\u0026rdquo; chapter, we collect various monitoring practice experiences, which will be categorized by different components. If you have good practical experience with a certain component, you are welcome to submit a PR and attach the link of your article to the corresponding component directory.\n How to monitor Java programs? There are usually the following methods:\n Actuator Micrometer JMX Components exposing metrics themselves Actuator # If you have a Spring Boot app, you can directly use Actuator to expose metrics. You can use the following prompt to ask GPT:\nHow to enable actuator and expose monitoring metrics in Prometheus format? I have pasted the AI\u0026rsquo;s reply here, which is probably correct. You can try it. If you find any problems, please feel free to submit an issue or PR.\nTo enable Actuator and expose Prometheus monitoring metrics in a Spring Boot project, you need to complete the following steps:\n1. Add dependencies # Add the following dependencies to your pom.xml:\n\u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-starter-actuator\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;io.micrometer\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;micrometer-registry-prometheus\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; If you are using Gradle:\nimplementation 'org.springframework.boot:spring-boot-starter-actuator' implementation 'io.micrometer:micrometer-registry-prometheus' 2. Configure application.properties or application.yml # Ensure that Actuator endpoints and the Prometheus metrics endpoint are exposed. For example:\nmanagement.endpoints.web.exposure.include=* management.endpoint.prometheus.enabled=true management.metrics.export.prometheus.enabled=true Or in YAML format:\nmanagement: endpoints: web: exposure: include: \u0026quot;*\u0026quot; endpoint: prometheus: enabled: true metrics: export: prometheus: enabled: true 3. Start the application # After starting your Spring Boot application, Actuator\u0026rsquo;s Prometheus endpoint will be exposed at /actuator/prometheus by default.\n4. Verify the metrics endpoint # Access it using a browser or curl:\nhttp://localhost:8080/actuator/prometheus You will see monitoring metrics in Prometheus format, which can be scraped using Prometheus or Categraf.\nNote:\n If you want to expose only some endpoints, such as prometheus and health, you can adjust the configuration to management.endpoints.web.exposure.include=prometheus,health For more advanced configurations (such as custom metrics), you can inject MeterRegistry in the code and register custom metrics. Micrometer # If you are not using a Spring Boot app, you can also use Micrometer to expose monitoring metrics in Prometheus format. Micrometer is an independent library that can be integrated
1with any Java application. Actuator is actually based on Micrometer. You can use the following prompt to ask GPT:\nIf actuator is not used and only micrometer is used, how to export metrics to Prometheus? I have pasted the AI\u0026rsquo;s reply here, which is probably correct. You can try it. If you find any problems, please feel free to submit an issue or PR.\n1. Add Micrometer and Prometheus dependencies # Taking Maven as an example:\n\u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;io.micrometer\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;micrometer-core\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;io.micrometer\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;micrometer-registry-prometheus\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; 2. Register PrometheusMeterRegistry # In your application code, manually create and register PrometheusMeterRegistry, for example:\nimport io.micrometer.prometheus.PrometheusMeterRegistry; import io.micrometer.prometheus.PrometheusConfig; import io.micrometer.core.instrument.MeterRegistry; PrometheusMeterRegistry prometheusRegistry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT); // You can register custom metrics through MeterRegistry // prometheusRegistry.counter(\u0026quot;my_custom_counter\u0026quot;).increment(); 3. Expose Prometheus metrics HTTP interface # Micrometer does not automatically expose an HTTP interface. You need to implement an HTTP endpoint yourself, such as using Spring MVC or other web frameworks:\nimport org.springframework.web.bind.annotation.GetMapping; import org.springframework.web.bind.annotation.RestController; @RestController public class PrometheusController { private final PrometheusMeterRegistry prometheusRegistry; public PrometheusController(PrometheusMeterRegistry prometheusRegistry) { this.prometheusRegistry = prometheusRegistry; } @GetMapping(\u0026quot;/prometheus\u0026quot;) public String scrape() { return prometheusRegistry.scrape(); } } Alternatively, if it is not a Spring project, you can use a web server like Jetty, Undertow, or Netty to directly expose the /prometheus path and use the content of prometheusRegistry.scrape() as the response. Finally, use Prometheus or Categraf to scrape this endpoint.\nJMX # If the Java program you want to monitor is not a self-developed program but an open-source component, such as Tomcat, Kafka, Zookeeper, etc., these components usually expose monitoring metrics through JMX. You can use the following prompt to ask GPT:\nHow to collect monitoring metrics for ordinary Java middleware such as Tomcat and Kafka? It is recommended to use JMX Exporter, which is a jar package that runs as a javaagent. Download the jmx_exporter jar package. Find the component\u0026rsquo;s startup command and add javaagent-related parameters to the startup parameters, such as -javaagent:/path/to/jmx_prometheus_javaagent-\u0026lt;version\u0026gt;.jar=PORT:/path/to/config.yaml to specify the path of the jmx_exporter jar, the port to expose metrics, and the path of the configuration file. This will expose Prometheus metrics on the specified port (e.g., http://localhost:PORT/metrics). Then use Prometheus or Categraf to scrape this endpoint.\nHowever, note that different components require different configuration files. JMX Exporter provides many examples at the specific address: https://github.com/prometheus/jmx_exporter/tree/main/examples. You can continue to ask AI:\n What do the configuration items in config.yaml mean? What is a Java MBean? How to configure JMX Exporter to collect a specific MBean? Throw the sample configuration provided by jmx_exporter to AI and let AI help you analyze the specific meaning of this configuration. Components exposing metrics themselves # Some components have built-in ways to expose monitoring metrics. For example, Tomcat can display various monitoring metrics on the HTTP endpoint /manager/status/all. The Tomcat collection plugin provided by Categraf collects monitoring metrics based on this endpoint. The configuration method is:\n Modify tomcat-users.xml and add the following content, which is equivalent to creating a user to access the /manager/status/all endpoint: \u0026lt;role rolename=\u0026quot;admin-gui\u0026quot; /\u0026gt; \u0026lt;user username=\u0026quot;tomcat\u0026quot; password=\u0026quot;s3cret\u0026quot; roles=\u0026quot;manager-gui\u0026quot; /\u0026gt; Comment out the following content in the file webapps/manager/META-INF/context.xml: \u0026lt;Valve className=\u0026quot;org.apache.catalina.valves.RemoteAddrValve\u0026quot; allow=\u0026quot;127\\.\\d+\\.\\d+\\.\\d+|::1|0:0:0:0:0:0:0:1\u0026quot; /\u0026gt; Configure the Tomcat collection address and authentication information in Categraf\u0026rsquo;s conf/input.tomcat/tomcat.toml. Note: The JMX method is universal, but the way each component exposes metrics varies. The above is just an example with Tomcat. For other components, you need to refer to their respective documents.\n "}),e.add({id:30,href:"/docs/usecase/alerting/",title:"Alerting Principles and Process Explanation",description:"The most important function of Nightingale monitoring is the alert
1ing engine. This article introduces the alerting principles and data flow of Nightingale, and explains all related functions involved in the entire alerting process.",content:"The key feature of Nightingale monitoring is its alerting engine. To achieve flexibility, the entire alerting process involves multiple functional points. This article introduces relevant knowledge from the perspective of principles and data flow, which will be helpful for you to use Nightingale and troubleshoot alerting issues.\nOverview of Data Flow Principles # Users configure alert rules in the Web UI, and the rules are stored in the DB (usually MySQL). The alerting engine (the n9e process has a built-in alerting engine, and the n9e-edge process in edge mode also has a built-in alerting engine) synchronizes alert rules from the DB to memory (usually n9e-edge cannot read the DB directly, but obtains alert rules by calling the interface of the central n9e). The alerting engine creates a goroutine (coroutine, which can be roughly understood as a lightweight thread) for each alert rule, periodically queries the storage according to the frequency configured in the alert rule, judges data anomalies, and finally generates alert events. After an alert event is generated, it is first persisted to the DB (usually MySQL), and then proceeds to the subsequent notification rules. Notification rules include two parts: several event processors (such as relabel, event update, event drop, ai summary, etc.), and several alert notification configurations (for example, Critical alert events are associated with phone calls and SMS notification media, while Warning alert events are only associated with email media). Alert Rules # The core of an alert rule is to configure a query condition. For example, for Prometheus data sources, PromQL is configured, and for ClickHouse data sources, SQL is configured. Then a threshold is set (in the Prometheus scenario, the threshold is included in PromQL and does not need to be configured separately). An alert will be triggered when the threshold is reached and the duration condition is met.\nThe alerting engine creates a goroutine (coroutine) for each rule, periodically queries the data source, and judges whether the alert conditions are met. Taking the Prometheus data source as an example, its principle is:\n Nightingale periodically calls the /api/v1/query interface of the data source, passing the current time and PromQL as query conditions to this interface. If the data source returns multiple records, it is likely that multiple alert events will be generated. Next, the duration needs to be considered. If the duration is 0, the alert event is generated immediately. If the duration is greater than 0, this record will be put into a cache, and the alert event will be generated only when the duration condition is met. During the duration, if the data is not found in subsequent execution cycles, this record will be deleted from the cache, and no alert event will be generated. A common problem here is that the alerting engine does not find data during the query, so it cannot generate an alert event, but later investigation finds that there was data meeting the threshold at that time, which is puzzling. There may be two reasons for this situation:\n It is caused by the delay in reporting monitoring data. Here, Nightingale is just a client, and the data source is the server. If the data source does not return data, you need to check the server side to see why the data is not returned, which is usually due to various delays in the data. The query timed out. Relevant logs can usually be seen in the log file. You can increase the query timeout time on the data source configuration page, or check why the data source returns slowly. In addition, there may be hardware problems, such as network card packet loss between the client and the server. For timeout logs, you can search for the keyword: alert-${datasource-id}-${alert-rule-id} Among them:\n ${datasource-id} is the ID of the data source, which can be seen on the data source details page. ${alert-rule-id} is the ID of the alert rule, which can be seen in the URL when editing the alert rule. When troubleshooting alert issues, first check whether an alert event is generated. If an alert event is generated, it means the alert rule is fine, and then check the subsequent notification-related issues. If no alert event is generated, it is a problem with the alert rule and data source. First, confirm the configuration of the alert rule before considering other things.\nEvent Persistence # After an alert event is generated, it needs to be written to the DB (usually MySQL), so that you can see this event in the alert event list. Sometimes the writing may fail, and if it fails, it is usually reflected in the logs. You can check the WARNING and ERROR logs.\nAssociating Alert Rules with Notification Rules # After an alert event is generated, which subsequent notification rule should it follow? That is, how to establish an association between alert rules and notification rules? There are two ways to establish the association:\n Configure notification rules directly in the alert rule. That is, all alert events generated by this alert rule will follow these notification rules. Do not configure notification rules in the alert rule, but configure subscription rules, that is: filter alert events according to various conditions in the subscription rule, and the filtered alert events will follow the notification rules configured in the subscription rule. Both methods are acceptable. The former is more intuitive, and it is recommended to use the former if there are no special requirements. However, for some global event processing, for example, if you want all alert events generated in Nightingale monitoring to go through a Callback processor, you can use a subscription rule to subscribe to all alert events, uniformly associate a global notification rule, and configure the Callback processor in this global notification rule.\nNotification Rule Configuration # The following figure is the editing page of notification rules, where I have marked the functions of each block:\nMost form items have a small question mark icon next to their titles. Hovering the mouse over it will display prompt information, which you can refer to for c
1onfiguration.\n ð¡ This page contains some notification test buttons. After clicking, you can select the generated alert events to test the notification rules, which is convenient for you to quickly verify whether the notification rules meet expectations. In addition, it should be noted that alert event persistence occurs before the notification rule, so each event processor in the notification rule will not modify the alert events in the DB.\n Event Processors # ð¡ The event Pipeline does not have a separate menu entry. As part of the notification rule, you can click the small gear icon in the \u0026ldquo;Event Processing\u0026rdquo; block on the notification rule editing page to expand the event processor configuration sidebar.\n Event processors are an advanced mechanism that allows you to perform various operations on alert events, such as:\n Relabel alert events, split some labels, modify some labels, etc. Update alert events: Nightingale sends the alert event to a third-party (such as CMDB) interface, and the third party can modify the alert event and return the modified content to continue the subsequent event processing logic, facilitating integration with external systems. Drop alert events: Some alert events do not need to be notified, and complex judgments can be made here to drop those that meet the conditions. Generate AI summaries: Send alert events to DeepSeek, etc., let AI help generate summaries and solutions, put the AI-generated content into the event, and send it out through notification media later. Two concepts to note here:\n The event processing Pipeline is the sidebar expanded by clicking the button next to \u0026ldquo;Notification Rule - Event Processing\u0026rdquo;, which contains the list of Pipelines. Each Pipeline can contain multiple Processors. To improve reusability, you can also simply have each Pipeline contain only one Processor. Each processor has a documentation link on the page. Clicking it will display detailed documentation. You can also refer to the materials in the following two links:\n Event Processor Description Custom Notification Media Open Source Nightingale Monitoring Implements Alert Silence During Release Notification Configuration # This part has been explained earlier, so it will not be repeated here. Please refer to:\n Design Purpose and Usage Instructions of Notification Rules "}),e.add({id:31,href:"/docs/usecase/bizgroup/",title:"Business Groups",description:"The concept of business groups in Nightingale monitoring, how to divide business groups, and the related design intentions. The division of business groups is similar to the service tree; usually, the top layer is organizational structure modeling, the middle layer is system service modeling, and the bottom layer is cluster division.",content:"Nightingale needs to manage many things, such as: alert rules, shielding rules, subscription rules, self-healing scripts, dashboards. When creating these things, you must first select a business group because these things must belong to a certain business group. The same goes for machines. After installing categraf, categraf will automatically register machine information with Nightingale. At this time, the machine will appear in the list of ungrouped machines. Administrators need to assign the machine to a certain business group so that members of the business group can use it.\nBusiness groups are frequently used in Nightingale. This article explains the concept of business groups in Nightingale monitoring and related design intentions.\nSource of Demand # Nightingale needs to manage many things, such as: alert rules, shielding rules, subscription rules, self-healing scripts, machines, etc. If all these things are placed in a single table for everyone to view and manage, it will be quite chaotic, so a mechanism for classification is needed.\nTherefore, we introduced the concept of \u0026ldquo;business groups\u0026rdquo;. A business group is a grouping mechanism. For example, DBAs can put MySQL alert rules in one business group (let\u0026rsquo;s call it DBA/MySQL) and Postgres alert rules in another business group (let\u0026rsquo;s call it DBA/Postgres); for example, Kubernetes operation and maintenance personnel split Kubernetes hosts by cluster, and place machines from different cluster
1s in different business groups, such as K8S/ClusterA, K8S/ClusterB, etc.\nEvolution # In early versions of Nightingale, business groups were displayed as a flat list. Later, it was found that business groups actually need a hierarchical structure. For example, the four business groups mentioned above:\n DBA/MySQL DBA/Postgres K8S/ClusterA K8S/ClusterB It is more convenient to view them when rendered as a tree structure:\nDBA âââ MySQL âââ Postgres K8S âââ ClusterA âââ ClusterB Therefore, in the new version of Nightingale, for compatibility with older versions, business groups are still stored in the database as a flat list, but in the front-end display, they can be rendered as a tree structure according to the delimiter in the name. For example, in the above example, the delimiter in the name is /, but you can also use other delimiters such as -, _, etc. In Nightingale\u0026rsquo;s menu System Configuration - Site Settings, you can set the business group display mode and delimiter.\n ð¢ It is recommended to use / as the delimiter.\n Issues # In Nightingale, business groups are globally shared. You can attach rules to business groups and also attach machines to business groups. This has the advantage of facilitating the reuse of business groups, that is, once a business group is created, it can be used in multiple places.\nHowever, this approach has a problem: different things have different granularities of grouping. For example, when grouping machines, we may divide them more finely, such as:\n DBA/MySQL/Proxy/RegionA DBA/MySQL/Proxy/RegionB But when grouping alert rules, dashboards, etc., we may not divide them as finely. For example, all dashboards of DBAs may be placed directly under DBA.\nThis problem is currently difficult to solve unless business groups are not made globally reusable. Machines have their own grouping, alert rules have their own grouping, and dashboards have their own grouping. This will lead to the need to create some business groups repeatedlyâafter creating them for machines, you have to create them again for alert rules. You can\u0026rsquo;t have both.\nBest Practices for Dividing Business Groups # Although business groups can be used for classifying various rules and grouping machines, their granularities are different. Usually, the granularity of machine grouping is finer, while that of rule grouping is coarser. Generally, plan business groups according to machine grouping first, and then attach various rules, dashboards, etc., to the middle-level business groups, which is almost sufficient. Let\u0026rsquo;s first talk about machine grouping.\nI wonder if readers have heard of the concept of \u0026ldquo;service tree\u0026rdquo;. The division of business groups is similar to the \u0026ldquo;service tree\u0026rdquo;. Generally speaking, the top layer is the modeling of the organizational structure, that is, the top-level nodes are information such as departments, businesses, teams, etc. For example:\n Infrastructure/Ops/Container Cloud refers to the container cloud team in the operation and maintenance team Infrastructure/Ops/Database refers to the database operation and maintenance team in the operation and maintenance team XBU/Business1/Product1 refers to the Product1 team under Business1 of a certain BU XBU/Business1/Product2 refers to the Product2 team under Business1 of a certain BU If the company\u0026rsquo;s organizational structure is relatively flat, the hierarchy of this information will be fewer. If the company\u0026rsquo;s organizational structure has many levels, more levels can be added.\nThe top layer is the modeling of the organizational structure, and the middle layer is the modeling of system services. Usually, the middle layer is divided into two layers: system-module. For example, Kubernetes is a system, and apiserver, etcd, scheduler, etc., are different modules in it.\nFinally, the bottom layer of business groups is usually divided by clusters. For example, if the quantity is really large, the concept of region can be introduced above the cluster. If the quantity is not that large, it is only necessary to divide by cluster. If there is only one cluster, it is even possible to cancel the bottom cluster node.\nTherefore, the business groups of the final container cloud platform may be divided as follows:\n Infrastru
1cture/Ops/Container Cloud/KubeUI/Webapi/South China Cluster Infrastructure/Ops/Container Cloud/KubeUI/Webapi/North China Cluster Infrastructure/Ops/Container Cloud/KubeUI/Report/South China Cluster Infrastructure/Ops/Container Cloud/KubeUI/Report/North China Cluster Infrastructure/Ops/Container Cloud/Kubernetes/etcd/South China Cluster Infrastructure/Ops/Container Cloud/Kubernetes/etcd/North China Cluster Infrastructure/Ops/Container Cloud/Kubernetes/apiserver/South China Cluster Infrastructure/Ops/Container Cloud/Kubernetes/apiserver/North China Cluster Infrastructure/Ops/Container Cloud/Kubernetes/scheduler/South China Cluster Infrastructure/Ops/Container Cloud/Kubernetes/scheduler/North China Cluster Infrastructure/Ops/Container Cloud/Kubernetes/node/South China Cluster Infrastructure/Ops/Container Cloud/Kubernetes/node/North China Cluster After business groups are divided according to the granularity of machines, rules and the like should be attached to the upper layers. For example, for Webapi alert rules, a special business group Infrastructure/Ops/Container Cloud/KubeUI/Webapi-Rules can be created, and then Webapi alert rules can be attached to this business group.\nIf there are few alert rules for Webapi and Report, it is also possible to directly create a Infrastructure/Ops/Container Cloud/KubeUI-Rules business group and attach all alert rules for Webapi and Report to this business group.\nThere are usually even fewer dashboards. You can directly create a Infrastructure/Ops/Container Cloud-Dashboards business group and attach all dashboards of the container cloud team to this business group.\nOf course, the above division logic is just a reference and cannot be applied to all companies. For example, some enterprises mainly use Nightingale to monitor a large number of devices (scattered in different regions and factories) rather than services. In this case, when dividing business groups, you can consider dividing them according to dimensions such as regions and factories.\nFAQ # Can\u0026rsquo;t Find the Business Group Operation Entry # Question: In the new V8 version, why can\u0026rsquo;t I find the business group menu under the personnel organization? How to add, delete, modify, and query business groups?\nAnswer: Business groups appear under many function menus, such as alert rules, shielding rules, dashboards, self-healing scripts, etc. You can directly add, delete, modify, and query business groups in these places, and there is no need to go to a separate business group menu. Move the mouse to the business group, an edit icon will automatically appear. Click the edit icon to edit or delete the business group. There is a small plus icon on the business group, which can be used to add a new business group.\n"}),e.add({id:32,href:"/docs/usecase/mute/",title:"Muting Rules",description:"Muting rules in Nightingale monitoring, how to configure them, and their design philosophy. Muting rules can block alert events to prevent them from disturbing users.",content:"Muting rules in Nightingale monitoring (menu entry: Alerts - Rule Management - Muting Rules TAB) are typically used in the following scenarios:\n To pre-block expected alerts, typically during maintenance activities, such as restarting a machine and pre-muting alerts related to that machine in advance For issues that cannot be fixed immediately but are already known; continuous alert notifications are unnecessary, so temporary muting is applied Principle # After an alert event is generated by the alert engine, it will first go through the judgment of muting rules before being persisted to the database. If it matches a muting rule, it will not be persisted to the database, let alone notify users. The working timing is as shown in the following figure:\nA muting rule is essentially a set of filter conditions used to filter the alert events that need to be muted. The filtering is based on the attributes and labels of the alert events. For example:\n Which data source the event comes from The severity level of the event The labels of the event Here is an example:\n Data source type: Prometheus; only alert events with the data source type Prometheus will be muted Data source: Not configured, meaning no restriction Event level: All three levels are checked, indicating that alert events of all levels will be muted Event labels: Two labels are configured, which is equivalent to: ident in (\u0026quot;10,1.2.3\u0026quot;, \u0026quot;10.1.2.4\u0026quot;) and rulename =~ \u0026quot;downtime\u0026quot; All the above filter conditions are in an and relationship, meaning an event will be muted only if it meets all the conditions.\nFAQ # 1. Why can I still see related alert events even after configuring a muting rule?\nThis is usually because the event was generated before the muting rule was configured. Muting rules are a post-remedial measure and cannot affect events that have already been generated.\n2. Multiple conditions in event labels are also in an and relationship, but users may not understand this\nAs shown in the following figure, the user configured two entries in the event label filtering, both with the label key ident:\nThe user intended to mute either of the two machines 10.1.2.113 and 10.1.2.114, but contrary to expectations, the relationship here is and, which is equivalent to: ident = \u0026quot;10.1.2.113\u0026quot; and ident = \u0026quot;10.1.2.114\u0026quot;. Obviously, this condition will never match any event. In fact, the user should use the in operator, as shown below:\n3. The effective scope of a muting rule is limited to the current business group\nThis is actually prompted on the page. To avoid misoperations, the effective scope of a muting rule is limited to the current business group. That is, a muting rule can only mute alert events under the current business group, and alert events under other business groups will not be affected.\nIn other words: If a muting rule and an alert rule belong to different business groups, the muting rule will not take effect on the alert rule.\nIf a muting rule took effect globally, it would be dangerous. For example, if a user arbitrarily configured an alert rule with filter conditions that could match all alert events, all alert events of the company would be muted.\n"}),e.add({id:33,href:"/docs/usecase/subscribe/",title:"Subscription Rules",description:"Explaining subscription rules in Nightingale monitoring, applicable to special alert notification scenarios such as global callbacks, fine-grained notification control, etc.",content:"The subscription rules in Nightingale monitoring can be accessed through the menu: Alerts - Rule Management - Subscription Rules TAB.\nWhy This Design # In Nightingale\u
10026rsquo;s alert rules, you can directly configure notification rules, which is very intuitive. Alert events generated by this alert rule will follow this notification rule. Datadog and Open-Falcon have similar designs, which are basically sufficient. However, if you are familiar with Zabbix and Prometheus, you will find that after an alert event is generated, who it is sent to actually follows a subsequent subscription logic:\n In the alert rule, only query conditions, thresholds, etc., are defined. That is, the alert rule is only responsible for event generation. As for how to notify and who to notify, the alert rule does not care about these. Users use the subscription mechanism to filter all alert events, and for these filtered alert events, specify relevant notification rules (who to notify, how to notify). This method is actually more flexible, but the disadvantage is that it is not intuitive enough. What about Nightingale? Both methods are supported. For ordinary users, it is recommended to use the method of \u0026ldquo;directly configuring notification rules in the alert rule\u0026rdquo; first, and use \u0026ldquo;subscription rules\u0026rdquo; for relatively rare scenarios, such as:\n My service depends on other services that I don\u0026rsquo;t manage (the alert rules of these services notify their responsible persons, not me). However, if these services fail, they may affect my service. So I want to subscribe to SLI-related alert events of these services (this is a demand scenario mentioned by some community users. Although it is written here, the author does not actually endorse it. Please consider it carefully. The author believes that each service should have a dashboard that lists SLI data of other dependent services. When your own service fails, you should check this dashboard to determine whether it is a problem with your own service or a dependent downstream service). Alert events generated by some general alert rules need to be distributed to different people. In this case, notification rules cannot be directly bound in the alert rules, so subscription rules can be used together to achieve this. Some global operations, such as global callbacks, can be implemented through subscription rules. For example, you want any alert event generated by the system to call back to a certain Webhook address. In this case, you can configure a global subscription rule to match all alert events, and then configure a Webhook notification rule. ð¡ Please carefully read the above text to understand the original intention of the subscription rule design. It is very, very, very important.\n Configuration Method # A subscription rule includes three parts of configuration:\n Name: The name of the subscription rule. It is recommended to use a meaningful name so that others can know what this subscription rule is for at a glance, which is convenient for maintenance. Filter configuration: Filter alert events in various dimensions. Note that it is to filter alert events, and these filtered alert events will follow the notification rules below. Notification rules: The filtered alert events will follow these notification rules. The overall logic is relatively clear. There are many configuration items in the filter configuration, which are introduced one by one below.\n Data source type: Used to filter which data source type the alert event is generated through. Data source: Used to filter which data source the alert event is generated through. Event level: Used to filter the level of alert events. Multiple levels can be selected. By default, all are selected, which is equivalent to severity in (\u0026quot;Info\u0026quot;, \u0026quot;Warning\u0026quot;, \u0026quot;Critical\u0026quot;). Selecting all is actually equivalent to not filtering in the \u0026ldquo;event level\u0026rdquo; dimension. Subscription alert rule: Used to filter which alert rule the alert event is generated by. Business group: Used to filter which business group the alert event belongs to. An alert event must be triggered by a certain alert rule, so the business group of the alert event is the business group that the alert rule belongs to (the current version is v8.0.0, and this part will be considered for optimization in the future. In the future, the business group of the machine in the alert event will also be c
1onsidered). Event label: Used to filter the labels of alert events. Pay attention to the usage of operators. The specific explanation is below. Subscription event duration: There is a small question mark icon on the right, which provides instructions for using this function, so it will not be repeated here. The above filter conditions are in an and relationship as a whole. The event label part can be configured with multiple filter items, and the relationship between different items is also and. If you want to match multiple label values, you can use the in operator or the regular expression =~.\nThe specific explanations for the operators are as follows:\n == matches a specific label value and can only fill in one. If you want to match multiple at the same time, you should use the in operator. =~ fills in a regular expression to flexibly match label values. in matches multiple label values, similar to in in SQL. not in does not match the label values, and multiple can be filled in, similar to not in in SQL, used to exclude multiple label values. != is not equal to, used to exclude a specific label value. !~ does not match the regular expression. Fill in a regular expression, and label values that match this regular expression will be excluded, similar to !~ in PromQL. Scenario example: Subscribe to all time-series alerts # For example, I want to subscribe to all time-series indicator-related alerts and then follow a unified Webhook notification rule for some automated processing logic. In this case, you can configure the data source type as Prometheus, select all event levels, and not configure other filter conditions.\n"}),e.add({id:34,href:"/docs/usecase/processor/",title:"Event Processor",description:"Nightingale monitoring system uses Event Processors for automated event handling. It can integrate with third-party systems to automatically process alert events.",content:"Event Processor is a concept introduced in Nightingale v8. After an alert event is generated, before sending notifications, Event Processors can be used to perform additional processing on the alert event. The open-source version supports 5 types of processors: Relabel, Callback, Event Update, Event Drop, and AI Summary. Different processors can form a Pipeline to perform a series of flexible processing on alert events. Examples of scenarios include:\n Integrating with internal CMDB to attach more abundant information to alert events Calling DeepSeek\u0026rsquo;s API to perform intelligent analysis on alert events and attach the analysis results to the alert events Sending all alert events to your own system, which is equivalent to mirroring a copy for subsequent analysis and processing Dropping some specific alert events, such as some recovery events that you don\u0026rsquo;t want to send notifications for Several concepts are involved here: notification rules, event processors, and event processing pipelines. A brief explanation:\n A notification rule can configure multiple event processing pipelines (Pipeline), which are executed in sequence An event processing pipeline (Pipeline) can configure multiple event processors (Processor), also executed in sequence As can be seen from the above screenshot, the entry menu is in Notifications - Rules. There may be many notification rules. When adding or editing a specific notification rule, you can see a configuration area for Event Processing, where you can reference multiple pre-created event pipelines (Pipeline). Where to add, delete, or modify event pipelines (Pipeline)? The entry is relatively hidden, in the small gear on the right side of Event Processing. Clicking it will expand a side panel, where you can add, delete, or modify event pipelines (Pipeline).\nWhen creating or editing an event pipeline (Pipeline), a new side panel will expand. In this new side panel, you can edit the Pipeline, and we can configure multiple event processors (Processor) in the Pipeline:\nClick Usage Instructions next to the event processor type field to view the usage documentation of the event processor.\nRelabel Processor # The Relabel processor is similar to the Relabel operation for monitoring metrics in Prometheus. However, in Nightingale, it is the Relabeling of alert events. Alert events also have label fields, and there is a need to process these labels, so the Relabel processor is provided here.\nFor specific usage instructions of the Relabel processor, click Usage Instructions next to the event processor type field on the Nightingale page to view them.\nCallback Processor # After an event is triggered, Nightingale can notify external systems through Callback, and external systems can perform automated processing based on the event content. For example, I have seen some companies develop their own alert notification systems. Instead of using Nightingale\u
10026rsquo;s notification mechanism, they directly send all alert events to their self-developed systems through the Callback processor.\nHere is a simple demonstration:\n First, create a \u0026ldquo;Notification Rule\u0026rdquo; because the Callback processor belongs to a certain Pipeline, and the Pipeline belongs to a certain notification rule. In the \u0026ldquo;Notification Rule\u0026rdquo;, reference the event processing Pipeline. The Pipeline needs to be created in advance (on the notification rule editing page, click the small gear on the right side of the processor to open the side panel, and create or edit the Pipeline in the side panel). The following screenshot is a detail page of a Pipeline, which contains a Callback processor. Here, http://10.99.1.107:8888/print is a test program of mine, which can print the received HTTP requests for easy demonstration. This program is also an open-source small program, and the address is github gohttpd.\n After creating the Pipeline, return to the notification rule page, and select the Pipeline just created in the event processing section.\nNext, you can configure the \u0026ldquo;Alert Rule\u0026rdquo; for testing to see if the generated alerts can be received by the third-party program.\nTo see the effect as soon as possible, you can create an alert rule that will definitely trigger the threshold, and then select the notification rule just created in the notification rule:\nWait a moment and observe whether the program http://10.99.1.107:8888/print receives the callback HTTP request. The results seen in my environment are as follows:\nAs can be seen from the above figure, the HTTP request contains information about the alert event, and its content is as follows:\n{ \u0026quot;id\u0026quot;: 1097371, \u0026quot;cate\u0026quot;: \u0026quot;prometheus\u0026quot;, \u0026quot;cluster\u0026quot;: \u0026quot;prom\u0026quot;, \u0026quot;datasource_id\u0026quot;: 1, \u0026quot;group_id\u0026quot;: 2, \u0026quot;group_name\u0026quot;: \u0026quot;DBA-Postgres\u0026quot;, \u0026quot;hash\u0026quot;: \u0026quot;54f5543591c6dc0e30139cae196a1eee\u0026quot;, \u0026quot;rule_id\u0026quot;: 54, \u0026quot;rule_name\u0026quot;: \u0026quot;test-callback\u0026quot;, \u0026quot;rule_note\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;rule_prod\u0026quot;: \u0026quot;metric\u0026quot;, \u0026quot;rule_algo\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;severity\u0026quot;: 2, \u0026quot;prom_for_duration\u0026quot;: 0, \u0026quot;prom_ql\u0026quot;: \u0026quot;cpu_usage_active{ident=\\\u0026quot;ulric-flashcat.local\\\u0026quot;} \\u003e 0\u0026quot;, \u0026quot;rule_config\u0026quot;: { \u0026quot;queries\u0026quot;: [{ \u0026quot;from\u0026quot;: 0, \u0026quot;prom_ql\u0026quot;: \u0026quot;cpu_usage_active{ident=\\\u0026quot;ulric-flashcat.local\\\u0026quot;} \\u003e 0\u0026quot;, \u0026quot;range\u0026quot;: { \u0026quot;display\u0026quot;: \u0026quot;now-undefineds to now-undefineds\u0026quot;, \u0026quot;end\u0026quot;: \u0026quot;now-undefineds\u0026quot;, \u0026quot;start\u0026quot;: \u0026quot;now-undefineds\u0026quot; }, \u0026quot;severity\u0026quot;: 2, \u0026quot;to\u0026quot;: 0, \u0026quot;unit\u0026quot;: \u0026quot;none\u0026quot; }] }, \u0026quot;prom_eval_interval\u0026quot;: 15, \u0026quot;callbacks\u0026quot;: [], \u0026quot;runbook_url\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;notify_recovered\u0026quot;: 1, \u0026quot;target_ident\u0026quot;: \u0026quot;ulric-flashcat.local\u0026quot;, \u0026quot;target_note\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;trigger_time\u0026quot;: 1749180264, \u0026quot;trigger_value\u0026quot;: \u0026quot;33.06867\u0026quot;, \u0026quot;trigger_values\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;trigger_values_json\u0026quot;: { \u0026quot;values_with_unit\u0026quot;: { \u0026quot;v\u0026quot;: { \u0026quot;value\u0026quot;: 33.06867479671808, \u0026quot;unit\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;text\u0026quot;: \u0026quot;33.07\u0026quot;, \u0026quot;stat\u0026quot;: 33.06867479671808 } } }, \u0026quot;tags\u0026quot;: [\u0026quot;__name__=cpu_usage_active\u0026quot;, \u0026quot;cpu=cpu-total\u0026quot;, \u0026quot;ident=ulric-flashcat.local\u0026quot;, \u0026quot;rulename=test-callback\u0026quot;], \u0026quot;tags_map\u0026quot;: { \u0026quot;__name__\u0026quot;: \u0026quot;cpu_usage_active\u0026quot;, \u0026quot;cpu\u0026quot;: \u0026quot;cpu-total\u0026quot;, \u0026quot;ident\u0026quot;: \u0026quot;ulric-flashcat.local\u0026quot;, \u0026quot;rulename\u0026quot;: \u0026quot;test-callback\u0026quot; }, \u0026quot;original_tags\u0026quot;: [\u0026quot;\u0026quot;, \u0026quot;\u0026quot;, \u0026quot;\u0026quot;, \u0026quot;\u0026quot;], \u0026quot;annotations\u0026quot;: {}, \u0026quot;is_recovered\u0026quot;: false, \u0026quot;last_eval_time\u0026quot;: 1749180264, \u0026quot;last_sent_time\u0026quot;: 1749180264, \u0026quot;notify_cur_number\u0026quot;: 1, \u0026quot;first_trigger_time\u0026quot;: 1749180264, \u0026quot;extra_config\u0026quot;: { \u0026quot;enrich_queries\u0026quot;: [] }, \u0026quot;status\u0026quot;: 0, \u0026quot;claimant\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;sub_rule_id\u0026quot;: 0, \u0026quot;extra_info\u0026quot;: null, \u0026quot;target\u0026quot;: null, \u0026quot;recover_config\u0026quot;: { \u0026quot;judge_type\u0026quot;: 0, \u0026quot;recover_exp\u0026quot;: \u0026quot;\u0026quot; }, \u0026quot;rule_hash\u0026quot;: \u0026quot;dc128d86d65326499bd03ecfbe56e4c3\u0026quot;, \u0026quot;extra_info_map\u0026quot;: null, \u0026quot;notify_rule_ids\u0026quot;: [3], \u0026quot;notify_version\u0026quot;: 0, \u0026quot;notify_rules\u0026quot;: null } The test is normal. If you have similar needs, you can use this Callback processor for integration and implement some automated logic in your program.\nEvent Update Processor # Among the event processors, there is also an Event Update processor, which is configured in the same way as Callback. Their working logic is also similar, with the following differences:\nWhen Nightingale calls the Callback address, it does not pay attention to the HTTP Response, but when calling Event Update, it will take the content of the HTTP Response as a new alert event for subsequent processing.\nTherefore, as the name suggests, Event Update is used to modify alert events. It is usually used to attach some additional information to alert events, such as:\n Submitting the event to AI for analysis, obtaining some conclusionary information, and attaching it to the event Querying some metadata from CMDB and attaching it to the event Note that the structure of the alert event cannot be modified arbitrarily. For example, directly adding a field at the top level of the JSON will not be recognized by subsequent processes. It is usually recommended to attach new content to the annotations field. In the example of my Callback processor above, annotations is empty, so the data structure cannot be seen. Actually, annotations is a map structure where both the map key and map value are string types. When you want to attach content, you also need to follow this structure.\n Advanced users can also modify other fields of the Event, but you need to be clear about the impact of your modifications on subsequent processes. Ordinary users only need to attach content to the annotations field and then serialize the entire new Event into JSON and put it in the body of the HTTP Response.\n Event Drop Processor # Among the event processors, there is also an Event Drop processor, which, as the name implies, is used to discard alert events. For example:\n In some scenarios, although an alert event is generated, you do not want to proceed with the subsequent notification logic. In this case, you can use the Event Drop processor to discard the event. To drop some alert events, filtering must be done. Usually, filtering can be done using various fields such as tags, annotations, and levels. The filtering rules may be very complex. How can this function be designed to be so flexible? We came up with a slightly complex but extremely flexible method, which is that users directly configure a section of template using go template syntax. The template can reference alert events and use syntax such as if for filtering. As long as the final rendered result of this go template is true, the event will be discarded.\nFor specific usage instructions, click Usage Instructions next to the Event Drop event processor type field on the Nightingale page to view them.\nAI Summary Processor # The documentation for the AI Summary processor is also complete, as shown in the following figure:\nClick Usage Instructions next to the AI Summary event processor type field to view the usage documentation of the AI Summary processor. There is a small question mark icon on the right side of each field below, and you can also see relevant prompt instructions when you move the mouse over it.\n"}),e.add({id:35,href:"/docs/usecase/media/",title:"Custom Notification Media",description:"Nightingale monitoring system supports custom notification media, which can send alarm events via HTTP, scripts, etc. This article introduces how to use WeCom (WeChat Work) applications as custom notification media to send alarm event notifications.",content:" Before reading this section, please ensure that you have read the content of the \u0026ldquo;Notification Rules\u0026rdquo; chapter, and also reviewed the external link materials mentioned in the \u0026ldquo;Notification Rules\u0026rdquo; chapter.\n Let\u0026rsquo;s simulate a scenario. Suppose I want to use a WeCom application (not the same as WeCom robot) for alarm notifications. Let\u0026rsquo;s go through the entire process.\nBasic Configuration # When sending notifications via a WeCom application, we need to know the WeCom account of the person being notified. Nightingale does not have this information by default. We can customize a contact field called wecomid, and each user can configure it themselves. In the image above, wecomid is the custom field name I created. Clicking \u0026ldquo;Contact Management\u0026rdquo; (only available to administrators) allows creating new contact methods. Here, I created a new contact method called wecomid to configure each user\u0026rsquo;s WeCom ID.\nCreate a custom notification medium corresponding to my own program. When a user configures Nightingale to send alarm messages to this notification medium, Nightingale will call my program, which will then call the WeCom API to send notification messages using the WeCom application (of course, this is just a demonstration; I still use the gohttpd applet introduced in the Event Processor chapter for demonstration). Explanation of several key fields for the notification medium above:\n Medium Type: Can be customized. I randomly named it wecomapp here. Notification media usually need to work with message templates. As long as the medium type of the message template is also wecomapp, the medium and message template can be associated. Contact Method: Select the wecomid contact method created earlier. This way, when Nightingale calls my program, it will pass the WeCom ID of the alarm recipient to me. URL: The address of my program. Nightingale will call this address via HTTP POST, and the content of the request body can be defined below. Request Body: Used to define the content of the callback HTTP request body, which can reference several variables. In this example, I referenced all three key variables. My request body:\n{ \u0026quot;events\u0026quot;: {{ jsonMarshal $events }}, \u0026quot;sendtos\u0026quot;: {{ jsonMarshal $sendtos }}, \u0026quot;tpl\u0026quot;: {{ jsonMarshal $tpl }} } $events is the list of alarm events to be sent. Although it\u0026rsquo;s a list, the open-source version will always have only one event. $sendtos is the list of recipients, which will eventually be a list of WeCom IDs. If the contact method configured here is Phone, $sendtos will be a list of phone numbers. $tpl is the content of the message template, which will be introduced immediately below. Message Template # Obviously, when finally calling the WeCom API to send alarm messages, we don\u0026rsquo;t want to send the entire event JSON, as users won\u0026rsquo;t be able to read it. We need to format the event for display (e.g., in markdown format), just like other notification media have corresponding message templates. Custom notification media also need message templates. Let\u0026rsquo;s create a message template:\nA message template can have multiple fields (for example, an email template needs custom subject and content, so it has two fields). However, in the WeCom application scenario, multiple fields are not needed. We can just create a content field, and later we can define the content of the content field using markdown format. For example:\nI also paste the content in markdown for your reference:\n**Rule Title**: {{$event.RuleName}} **Monitoring Metric**: {{$event.TagsJSON}} **Sending Time**: {{timestamp}} This is just a demonstration, so the fields rendered in markdown are relatively few. You can refer to other templates to enrich the content later.\n ð¡ Note that the medium type of the message template needs to be consistent with that of the notification medium to enable association. Therefore, I still used wecomapp here.\n Testing # Now we can test it. Create a notification rule:\nFor testing convenience, this notification rule has no filtering conditions configured, meaning any alarm event will be sent to the \u0026ldquo;WeCom Application\u0026rdquo; custom notification medium.\nFinally, we create an alarm rule and associate it with the notification rule just created:\nWait a moment and check if the http://10.99.1.107:8888/print program receives the callback HTTP request. The result in my environment is as follows:\nI paste the content of the HTTP request body for your reference:\n{ \u0026quot;events\u0026quot;: [{ \u0026quot;id\u0026quot;: 1097655, \u0026quot;
1cate\u0026quot;: \u0026quot;prometheus\u0026quot;, \u0026quot;cluster\u0026quot;: \u0026quot;prom\u0026quot;, \u0026quot;datasource_id\u0026quot;: 1, \u0026quot;group_id\u0026quot;: 2, \u0026quot;group_name\u0026quot;: \u0026quot;DBA-Postgres\u0026quot;, \u0026quot;hash\u0026quot;: \u0026quot;f75556af7cedbe250d3d8ab709634c96\u0026quot;, \u0026quot;rule_id\u0026quot;: 56, \u0026quot;rule_name\u0026quot;: \u0026quot;æµè¯èªå®ä¹éç¥åªä»2\u0026quot;, \u0026quot;rule_note\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;rule_prod\u0026quot;: \u0026quot;metric\u0026quot;, \u0026quot;rule_algo\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;severity\u0026quot;: 2, \u0026quot;prom_for_duration\u0026quot;: 0, \u0026quot;prom_ql\u0026quot;: \u0026quot;cpu_usage_active{ident=\\\u0026quot;ulric-flashcat.local\\\u0026quot;} \\u003e 0\u0026quot;, \u0026quot;rule_config\u0026quot;: { \u0026quot;queries\u0026quot;: [{ \u0026quot;from\u0026quot;: 0, \u0026quot;prom_ql\u0026quot;: \u0026quot;cpu_usage_active{ident=\\\u0026quot;ulric-flashcat.local\\\u0026quot;} \\u003e 0\u0026quot;, \u0026quot;range\u0026quot;: { \u0026quot;display\u0026quot;: \u0026quot;now to now\u0026quot;, \u0026quot;end\u0026quot;: \u0026quot;now\u0026quot;, \u0026quot;start\u0026quot;: \u0026quot;now\u0026quot; }, \u0026quot;severity\u0026quot;: 2, \u0026quot;to\u0026quot;: 0, \u0026quot;unit\u0026quot;: \u0026quot;none\u0026quot; }] }, \u0026quot;prom_eval_interval\u0026quot;: 15, \u0026quot;callbacks\u0026quot;: [], \u0026quot;runbook_url\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;notify_recovered\u0026quot;: 1, \u0026quot;target_ident\u0026quot;: \u0026quot;ulric-flashcat.local\u0026quot;, \u0026quot;target_note\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;trigger_time\u0026quot;: 1749196393, \u0026quot;trigger_value\u0026quot;: \u0026quot;33.06867\u0026quot;, \u0026quot;trigger_values\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;trigger_values_json\u0026quot;: { \u0026quot;values_with_unit\u0026quot;: { \u0026quot;v\u0026quot;: { \u0026quot;value\u0026quot;: 33.06867479671808, \u0026quot;unit\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;text\u0026quot;: \u0026quot;33.07\u0026quot;, \u0026quot;stat\u0026quot;: 33.06867479671808 } } }, \u0026quot;tags\u0026quot;: [\u0026quot;__name__=cpu_usage_active\u0026quot;, \u0026quot;cpu=cpu-total\u0026quot;, \u0026quot;ident=ulric-flashcat.local\u0026quot;, \u0026quot;rulename=æµè¯èªå®ä¹éç¥åªä»2\u0026quot;], \u0026quot;tags_map\u0026quot;: { \u0026quot;__name__\u0026quot;: \u0026quot;cpu_usage_active\u0026quot;, \u0026quot;cpu\u0026quot;: \u0026quot;cpu-total\u0026quot;, \u0026quot;ident\u0026quot;: \u0026quot;ulric-flashcat.local\u0026quot;, \u0026quot;rulename\u0026quot;: \u0026quot;æµè¯èªå®ä¹éç¥åªä»2\u0026quot; }, \u0026quot;original_tags\u0026quot;: [\u0026quot;\u0026quot;, \u0026quot;\u0026quot;, \u0026quot;\u0026quot;, \u0026quot;\u0026quot;], \u0026quot;annotations\u0026quot;: {}, \u0026quot;is_recovered\u0026quot;: false, \u0026quot;last_eval_time\u0026quot;: 1749196393, \u0026quot;last_sent_time\u0026quot;: 1749196393, \u0026quot;notify_cur_number\u0026quot;: 1, \u0026quot;first_trigger_time\u0026quot;: 1749196393, \u0026quot;extra_config\u0026quot;: { \u0026quot;enrich_queries\u0026quot;: [] }, \u0026quot;status\u0026quot;: 0, \u0026quot;claimant\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;sub_rule_id\u0026quot;: 0, \u0026quot;extra_info\u0026quot;: null, \u0026quot;target\u0026quot;: null, \u0026quot;recover_config\u0026quot;: { \u0026quot;judge_type\u0026quot;: 0, \u0026quot;recover_exp\u0026quot;: \u0026quot;\u0026quot; }, \u0026quot;rule_hash\u0026quot;: \u0026quot;3c73b1b7f98f4c5a0178c85dedabf008\u0026quot;, \u0026quot;extra_info_map\u0026quot;: null, \u0026quot;notify_rule_ids\u0026quot;: [4], \u0026quot;notify_version\u0026quot;: 0, \u0026quot;notify_rules\u0026quot;: null }], \u0026quot;sendtos\u0026quot;: [\u0026quot;qinxiaohui\u0026quot;], \u0026quot;tpl\u0026quot;: { \u0026quot;content\u0026quot;: \u0026quot;**è§åæ é¢**: æµè¯èªå®ä¹éç¥åªä»2 \\\\n**çæ§ææ **: [__name__=cpu_usage_active cpu=cpu-total ident=ulric-flashcat.local rulename=æµè¯èªå®ä¹éç¥åªä»2] \\\\n**åéæ¶é´**: 2025-06-06 15:53:13 \u0026quot; } }
1 This request body contains not only event details but also tpl, which is the rendered content. You can directly use this content to call the WeCom API. sendtos is a list of WeCom IDs. Since we configured the contact method in the WeCom application notification medium as wecomid, sendtos will be a list of WeCom IDs. If you configure the contact method in the notification medium as Phone, sendtos will be a list of phone numbers.\nWith all the data obtained, the next step is your custom logic and calling the WeCom API. Here, I used gohttpd just to demonstrate the content of the request body. Obviously, the gohttpd tool does not have the ability to send WeCom application messages. You need to implement this logic yourself: write a program that listens on an HTTP port, provides an HTTP POST interface (like gohttpd above), receives callback requests from Nightingale, parses the request body, retrieves event details, recipient lists, and message template content, and finally calls the WeCom API to send messages.\n"}),e.add({id:36,href:"/docs/usecase/sso/",title:"Single Sign-On (SSO)",description:"Nightingale monitoring supports Single Sign-On (SSO) functionality, allowing login to Nightingale monitoring via SSO. SSO supports multiple protocols such as OAuth2, OIDC, etc.",content:"Nightingale monitoring supports Single Sign-On (SSO) functionality, including multiple protocols such as LDAP, CAS, OAuth2, OIDC, etc. The SSO feature allows users to log in to Nightingale monitoring through a unified identity authentication system, simplifying user management and login processes while reducing security risks.\nFor CAS, OAuth2, and OIDC methods, after a user logs in to Nightingale via SSO, Nightingale will check if the currently logged-in user exists in Nightingale\u0026rsquo;s user table. If not, a user will be automatically created. If the user exists, Nightingale will overwrite the existing user information in Nightingale with the user information from SSO (provided that the configuration item CoverAttributes = true is set, as described later). This ensures that users only need to maintain their phone numbers and emails in SSO, and Nightingale will automatically synchronize this information during login (note that synchronization only occurs during login, so users must log in to Nightingale via SSO at least once; otherwise, their information will not exist in Nightingale).\nConfiguring OIDC # This is the most recommended method. If your SSO supports both OIDC and OAuth2, it is advisable to use OIDC.\nConfiguration Item Explanation # Below is an explanation of each OIDC configuration item:\n# Whether to enable OIDC single sign-on; Nightingale can enable multiple SSO methods simultaneously Enable = false # The login page will display a hyperlink to the SSO login address; DisplayName configures the text content of the hyperlink DisplayName = 'OIDC' # After OIDC login verification is passed, it needs to redirect to Nightingale. The following configures the callback address used by Nightingale for OIDC # You need to replace n9e.com with your Nightingale address; /callback is a fixed path RedirectURL = 'http://n9e.com/callback' # OIDC SSO server root address; replace with your OIDC server address SsoAddr = 'http://sso.example.org' # Logout address of the OIDC SSO server; when the user clicks logout in Nightingale, they will be redirected to this address to complete synchronous logout of Nightingale and SSO SsoLogoutAddr = 'http://sso.example.org/session/end' # ClientId and ClientSecret assigned to Nightingale by OIDC; must be configured, generally obtained when registering the application on the SSO server ClientId = '' ClientSecret = '' # When a user logs in to Nightingale via SSO and Nightingale finds that the user does not exist in Nightingale's user table, a user will be automatically created # When creating a user, the user needs to be assigned a role; the following configures the list of default roles DefaultRoles = ['Standard'] # When a user logs in to Nightingale via SSO, Nightingale will request user information from the SSO server and write the user information into Nightingale's user table # If CoverAttributes = true, it will over
1write existing user information in Nightingale, such as phone number, email, etc. Usually, this should be set to true CoverAttributes = true # Scope is a concept in the OIDC protocol, representing the list of user information fields requested; usually the following fields Scopes = ['openid', 'profile', 'email', 'phone'] # The user information fields in OIDC do not 100% correspond to those in Nightingale # Therefore, configure below: which fields in OIDC correspond to each field in Nightingale # Username, Nickname, Phone, Email are user information fields in Nightingale # The following sub, nickname, phone_number, email are user field names in OIDC # Adjust according to the user information fields of your OIDC server [Attributes] Username = 'sub' Nickname = 'nickname' Phone = 'phone_number' Email = 'email' Configuration Item FAQ # 1. Users can log in successfully using OIDC, but username, phone number, etc., cannot be obtained\nYou can adjust Nightingale\u0026rsquo;s log level to DEBUG (adjust in config.toml), then restart Nightingale, filter the log keyword: sso_exchange_user: oidc info, and test the login again. You can check which user information is obtained from the single sign-on system, then adjust the field mapping in Attributes according to the actual situation.\nAuthing Integration Demo # The following uses Authing as the OIDC SSO server for demonstration. First, create an application on Authing to obtain ClientId and ClientSecret.\nThen configure OIDC-related information in Nightingale:\nEnable = true DisplayName = 'OIDC' RedirectURL = 'http://192.168.127.151:17000/callback' SsoAddr = 'https://n9e.authing.cn/oidc' SsoLoginOutAddr = 'https://n9e.authing.cn/oidc/session/end' ClientId = '65befb5b452d4854f9731b9b' ClientSecret = '0af4...' CoverAttributes = true DefaultRoles = ['Standard'] Scopes = ['openid', 'profile', 'username', 'email', 'phone'] [Attributes] Username = 'username' Nickname = 'nickname' Phone = 'phone_number' Email = 'email' The above 192.168.127.151:17000 is the Nightingale address of my test environment; you need to replace it with your own Nightingale address.\nFeishu Integration Demo # Feishu also supports the OIDC protocol, and we will explain this method. Refer to Feishu\u0026rsquo;s official documentation to create an application: Configure Application Single Sign-On. Related configurations:\n Authorization mode: You can select both authorization_code and refresh_token Scope: You can select openid profile email phone offline_access Callback address: Fill in http://n9e.com/callback, note to replace n9e.com with your Nightingale address, which needs to be publicly accessible unless your Feishu is also deployed on a private intranet After configuration, you can obtain the Issuer (i.e., SSO Server address), ClientId, and ClientSecret, and configure them in Nightingale. In addition, you can also obtain the SSO Logout address, which is similar to:\nhttps://anycross.feishu.cn/sso/....../oidc/revoke Configure this address in Nightingale\u0026rsquo;s SsoLogoutAddr. Although this address is configured, simultaneous logout is not possible. The following is the official explanation from Feishu\u0026rsquo;s documentation:\n Since the login state of the SSO application is derived from the Feishu login state, single logout is not supported. That is, when logging out of the SSO application, Feishu cannot be logged out simultaneously. Although the platform provides a single logout address, this address is to prevent third-party systems from setting it as a required item, and the address itself does not take effect.\n The final configuration in Nightingale is as follows:\nEnable = true DisplayName = 'OIDC' RedirectURL = 'http://n9e.com/callback' SsoAddr = 'https://anycross.feishu.cn/sso/XXXXX' SsoLoginOutAddr = 'https://anycross.feishu.cn/sso/XXXXX/oidc/revoke' ClientId = 'xxx' ClientSecret = 'xxx' CoverAttributes = true DefaultRoles = ['Standard'] Scopes = ['openid', 'profile', 'email', 'phone'] [Attributes] Username = 'name' Nickname = 'name' Phone = 'phone_number' Email = 'email' The above n9e.com needs to be replaced with your own Nightingale address, which must be publicly accessible unless your Feishu is also deployed on a private intranet.\n Keycloak Integration Demo # A netizen previously wrote an article explaining how to integrate Grafana and Nightingale with Keycloak. You can refer to it: Deep Integration of Grafana and Nightingale with Keycloak.\nConfiguring OAuth2 # If your SSO supports both OIDC and OAuth2, it is recommended to use OIDC. Use OAuth2 only when necessary, as OAuth2 has more pitfalls.\nConfiguration Item Explanation # # Whether to enable OAuth2 single sign-on; Nightingale can enable multiple SSO methods simultaneously Enable = false # The login page will display a hyperlink to the SSO login address; DisplayName configures the text content of the hyperlink DisplayName = 'OAuth2' # After SSO login verification is passed, it needs to redirect to Nightingale. The following configures the callback address used by Nightingale for OAuth2 # You need to replace n9e.com with your Nightingale address; /callback/oauth is a fixed path RedirectURL = 'http://n9e.com/callback/oauth' # OAuth2 SSO server root address; replace with your OAuth2 server address SsoAddr = 'https://sso.example.com/oauth2/authorize' # Logout address of the OAuth2 SSO server; when the user clicks logout in Nightingale, they will be redirected to this address to complete synchronous logout of Nightingale and SSO SsoLogoutAddr = 'https://sso.example.com/oauth2/authorize/session/end' # Address to obtain OAuth2 token TokenAddr = 'https://sso.example.com/oauth2/token' # User information address provided by OAuth2; Nightingale will obtain user information through this address UserInfoAddr = 'https://sso.example.com/api/v1/user/info' # When obtaining user information from OAuth2, the token obtained in the previous step needs to be placed in the request header # The token can be placed in the header, formdata, or querystring, depending on the requirements of your OAuth2 server TranTokenMethod = 'header' # ClientId and ClientSecret assigned to Nightingale by OAuth2; must be configured, generally obtained when registering the application on the SSO server ClientId = '' ClientSecret = '' # When a user logs in to Nightingale via SSO and Nightingale finds that the user does not exist in Nightingale's user table, a user will be automatically created # When creating a user, the user needs to be assigned a role; the following configures the list of default roles DefaultRoles = ['Standard'] # When a user logs in to Nightingale via SSO, Nightingale will request user information from the SSO server and write the user information into Nightingale's user table # If CoverAttributes = true, it will over
1write existing user information in Nightingale, such as phone number, email, etc. Usually, this should be set to true CoverAttributes = true # Whether the JSON format data returned when obtaining user data from OAuth2 is an array, depending on the return format of your OAuth2 server # If it is an array, Nightingale will take the first element as user information UserinfoIsArray = false # Prefix for OAuth2 user information, usually 'data', meaning the returned JSON data contains a 'data' field with user information UserinfoPrefix = 'data' # Scope is a concept in the OAuth2 protocol, representing the list of user information fields requested; usually the following fields Scopes = ['profile', 'email', 'phone'] # The user information fields in OAuth2 do not 100% correspond to those in Nightingale # Therefore, configure below: which fields in OAuth2 correspond to each field in Nightingale # Username, Nickname, Phone, Email are user information fields in Nightingale # The following sub, nickname, phone_number, email are user field names in OAuth2 # Adjust according to the user information fields of your OAuth2 server [Attributes] Username = 'sub' Nickname = 'nickname' Phone = 'phone_number' Email = 'email' Authing Integration Demo # Use Authing as the OAuth2 SSO server for demonstration. First, enable OAuth2 on Authing. A configuration example is as follows:\nThen configure OAuth2-related information in Nightingale:\nEnable = true DisplayName = 'OAuth2' RedirectURL = 'http://192.168.127.151:17000/callback/oauth' SsoAddr = 'https://n9e.authing.cn/oauth/auth' SsoLogoutAddr = 'https://n9e.authing.cn/oauth/session/end' TokenAddr = 'https://n9e.authing.cn/oauth/token' UserInfoAddr = 'https://n9e.authing.cn/oauth/me' TranTokenMethod = 'header' ClientId = '65befb5b452d4854f9731b9b' ClientSecret = '0af4...' CoverAttributes = true DefaultRoles = ['Standard'] UserinfoIsArray = false UserinfoPrefix = '' Scopes = ['profile', 'username', 'email', 'phone'] [Attributes] Username = 'username' Nickname = 'nickname' Phone = 'phone' Email = 'email' The above 192.168.127.151:17000 is the Nightingale address of my test environment; you need to replace it with your own Nightingale address.\nConfiguring CAS # Nightingale monitoring also supports single sign-on via the CAS protocol. Compared to OIDC, it has more pitfalls, so use with caution.\nConfiguration Item Explanation # # Whether to enable CAS single sign-on; Nightingale can enable multiple SSO methods simultaneously Enable = false # The login page will display a hyperlink to the SSO login address; DisplayName configures the text content of the hyperlink DisplayName = 'CAS' # After CAS login verification is passed, it needs to redirect to Nightingale. The following configures the callback address used by Nightingale for CAS # You need to replace n9e.com with your Nightingale address; /callback/cas is a fixed path RedirectURL = 'http://n9e.com/callback/cas' # CAS SSO server root address; replace with your CAS server address SsoAddr = 'https://cas.example.com/cas' # Logout address of the CAS SSO server; when the user clicks logout in Nightingale, they will be redirected to this address to complete synchronous logout of Nightingale and SSO SsoLogoutAddr = 'https://cas.example.com/cas/session/end' # The LoginPath configuration item is to be compatible with different CAS versions, as login addresses may vary between versions # If you configure LoginPath, Nightingale will append LoginPath to SsoAddr as the login address # If you do not configure LoginPath, Nightingale's logic is: # 1. If SsoAddr contains the keyword p3, set LoginPath = '/login' # 2. If it does not contain the keyword p3, set LoginPath = '/cas/login' LoginPath = '' # When a user logs in to Nightingale via SSO and Nightingale finds that the user does not exist in Nightingale's user table, a user will be automatically created # When creating a user, the user needs to be assigned a role; the following configures the list of default roles DefaultRoles = ['Standard'] # When a user logs in to Nightingale via SSO, Nightingale will request user information from the SSO server and write the user information into Nightingale's user table # If CoverAttributes = true, it will over
1write existing user information in Nightingale, such as phone number, email, etc. Usually, this should be set to true CoverAttributes = true # The user information fields in CAS do not 100% correspond to those in Nightingale # Therefore, configure below: which fields in CAS correspond to each field in Nightingale # Username, Nickname, Phone, Email are user information fields in Nightingale # The following sub, nickname, phone_number, email are user field names in CAS # Adjust according to the user information fields of your CAS server [Attributes] Username = 'sub' Nickname = 'nickname' Phone = 'phone_number' Email = 'email' Authing Integration Demo # Use Authing as the CAS SSO server for demonstration. First, enable CAS on Authing. A configuration example is as follows:\nThen configure CAS-related information in Nightingale:\nEnable = true DisplayName = 'CAS' RedirectURL = 'http://192.168.127.151:17000/callback/cas' SsoAddr = 'https://n9e.authing.cn/cas-idp/65befb5b452d4854f9731b9b' SsoLogoutAddr = 'https://n9e.authing.cn/cas-idp/65befb5b452d4854f9731b9b/logout' LoginPath = '/login' CoverAttributes = true DefaultRoles = ['Standard'] [Attributes] Username = 'username' Nickname = 'nickname' Phone = 'phone_number' Email = 'email' The above 192.168.127.151:17000 is the Nightingale address of my test environment; you need to replace it with your own Nightingale address.\nConfiguring LDAP # Nightingale monitoring also supports authentication login via the LDAP protocol. LDAP is a lightweight directory access protocol, usually used for user authentication and authorization within enterprises. The previously mentioned SSO mechanisms (OIDC, OAuth2, CAS) cannot periodically synchronize all user information to Nightingale, but LDAP can achieve this.\nLDAP does not have a separate login hyperlink entry on the page. When a user enters a username and password to log in to Nightingale, Nightingale first queries the user information in the database. If not found, it automatically checks if LDAP is enabled. If enabled, it directly uses LDAP for authentication login.\nConfiguration Item Explanation # # Whether to enable LDAP single sign-on; Nightingale can enable multiple SSO methods simultaneously Enable = false # LDAP server address and port, TLS, StartTLS, etc. configurations # Configure according to your own environment Host = 'ldap.example.org' Port = 389 TLS = false StartTLS = true # Root DN of the LDAP server; you can Google or use GPT for more information BaseDn = 'dc=example,dc=org' # Administrator information; this account needs to have permission to query all user information BindUser = 'cn=manager,dc=example,dc=org' BindPass = '*******' # Whether to synchronize created users in LDAP to Nightingale SyncAddUsers = false # Whether to synchronize deleted user operations in LDAP to Nightingale SyncDelUsers = false # Synchronization frequency, unit: seconds SyncInterval = 86400 # Filter conditions to check if the user exists in LDAP during login # OpenLDAP and AD usually have different filter formats # The format for OpenLDAP may be: (\u0026amp;(uid=%s)) # The format for AD may be (\u0026amp;(sAMAccountName=%s)) # Adjust according to your LDAP server type AuthFilter = '(\u0026amp;(uid=%s))' # Filter conditions for querying all users in LDAP # Adjust according to your LDAP server type UserFilter = '(\u0026amp;(uid=*))' # When a user logs in to Nightingale via LDAP and Nightingale finds that the user does not exist in Nightingale's user table, a user will be automatically created # When creating a user, the user needs to be assigned a role; the following configures the list of default roles DefaultRoles = ['Standard'] # When a user logs in to Nightingale via LDAP, Nightingale will request user information from the LDAP server and write the user information into Nightingale's user table # If CoverAttributes = true, it will overwrite existing user information in Nightingale, such as phone number, email, etc. Usually, this should be set to true CoverAttributes = true # The user information fields in LDAP do not 100% correspond to those in Nightingale # Therefore, configure below: which fields in LDAP correspond to each field in Nightingale # Username, Nickname, Phone, Email are user information fields in Nightingale # The following uid, cn, mobile, mail are user field names in LDAP # Adjust according to the user information fields of your LDAP server [Attributes] Username = 'uid' Nickname = 'cn' Phone = 'mobile' Email = 'mail' If you are still unsure how to configure the above configuration information after reading the comments, you can consult your company\u0026rsquo;s LDAP administrator, who is likely to be familiar with it.\n"}),e.add({id:37,href:"/docs/usecase/api/",title:"API",description:"This article introduces how to use API to call the interfaces of Nightingale monitoring system, mainly including two types of interfaces: one is page operation interface, and the other is data push interface",content:"This article introduces how to use API to call the interfaces of Nightingale monitoring system. There are mainly two types of interfaces: one is page operation interface, which imitates user operations on the page through API, and the other is data push interface, for example, if your own program collects monitoring data and wants to push it to Nightingale.\nPage Operation Interfaces # Page operation interfaces mainly simulate user operations on the page, such as creating alarm rules, modifying machine tags, modifying machine remarks, adjusting the business group that the machine belongs to, etc. All operations that users can perform on the page can be completed through API, and you can use these interfaces to achieve automated o
1perations.\nObviously, there are two prerequisites for calling the API:\n Complete authentication Understand which interfaces are available and what parameters each interface has Complete Authentication # Here we directly explain the authentication method for versions above v8.0.0-beta.5, which is the personal center token method, and it is the simplest way.\n1. Modify the Configuration File # Modify Nightingale\u0026rsquo;s configuration file etc/config.toml, ensure that HTTP.TokenAuth is configured and Enable is set to true, as shown below:\n... [HTTP.RSA] OpenRSA = false [HTTP.TokenAuth] Enable = true [DB] ... 2. Obtain Token # Log in to Nightingale monitoring, click the avatar in the upper right corner, enter the personal information page, click the \u0026ldquo;Token Management\u0026rdquo; tab, then click \u0026ldquo;Create Token\u0026rdquo;, give it a random name, and you can get a Token.\n3. Use Token # When calling the API, you need to add the X-User-Token field in the Header of the HTTP request, and the value is the Token you just created. Example of calling API with cURL command:\ncurl -s -X GET \u0026quot;http://\u0026lt;NIGHTINGALE_HOST\u0026gt;\u0026lt;API_URL_PATH\u0026gt;\u0026quot; \\ -H \u0026quot;X-User-Token: \u0026lt;YOUR_TOKEN\u0026gt;\u0026quot; \\ -H \u0026quot;Content-Type: application/json\u0026quot; Let\u0026rsquo;s test the interface for obtaining personal information:\ncurl -s -H \u0026quot;X-User-Token: e6897d32-c237-4d27-a0fc-786345b682ea\u0026quot; -H \u0026quot;Content-Type: application/json\u0026quot; 'http://10.99.1.106:8003/api/n9e/self/profile' | python3 -m json.tool { \u0026quot;dat\u0026quot;: { \u0026quot;id\u0026quot;: 1, \u0026quot;username\u0026quot;: \u0026quot;root\u0026quot;, \u0026quot;nickname\u0026quot;: \u0026quot;Root\u0026quot;, \u0026quot;phone\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;email\u0026quot;: \u0026quot;[email protected]\u0026quot;, \u0026quot;portrait\u0026quot;: \u0026quot;/image/avatar1.png\u0026quot;, \u0026quot;roles\u0026quot;: [ \u0026quot;Admin\u0026quot; ], \u0026quot;contacts\u0026quot;: { \u0026quot;wecomid\u0026quot;: \u0026quot;qinxiaohui\u0026quot; }, \u0026quot;maintainer\u0026quot;: 0, \u0026quot;create_at\u0026quot;: 1733229739, \u0026quot;create_by\u0026quot;: \u0026quot;system\u0026quot;, \u0026quot;update_at\u0026quot;: 1749811268, \u0026quot;update_by\u0026quot;: \u0026quot;root\u0026quot;, \u0026quot;belong\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;admin\u0026quot;: true, \u0026quot;user_groups\u0026quot;: null, \u0026quot;busi_groups\u0026quot;: null, \u0026quot;last_active_time\u0026quot;: 1749811088 }, \u0026quot;err\u0026quot;: \u0026quot;\u0026quot; } As above, the content is returned normally, indicating success. If you want to write a program to call the API, you need to check:\n Whether the HTTP status code returned by Nightingale is 200. If the status code is not 200, it indicates that the request failed. At this time, you can print the Response Body to view the specific error information. If the status code is 200, the Response Body must be JSON. At this time, you also need to check whether the err field in the JSON data is empty. If it is not empty, there is a problem. Understand Available Interfaces # Open the Nightingale page directly with Chrome, press F12 to open the developer tools, switch to the Network tab, then operate on the page, such as creating an alarm rule or modifying machine tags. You will see many API requests in the Network, which are the API interfaces of Nightingale. For example:\n You can see the Request Method and URL under Headers You can see the returned content under Response The above interface does not have any Query string parameters. If there are Query string parameters, they are usually displayed in the URL. In addition, if it is a POST request, you need to study the format of the Request Body. At that time, a Payload tab will appear, and you can see the content format of the Request Body under Payload.\nThis method is much better than the interface document. The interface document is often forgotten to be updated, and the differences between different versions are often forgotten to be explained. However, by viewing Chrome in this way, the information is absolutely 100% accurate. It is clear which interfaces there are and what parameters they have. Every operation and maintenance personnel and back-end R\u0026amp;D should know this method.\nData Push Interfaces # There are usually two ways for your own program to expose monitoring data: one is to embed the Prometheus SDK, expose the /metrics interface, and then use Prometheus or Categraf to scrape (called PULL mode);
1 the other is to directly call Nightingale\u0026rsquo;s API interface to push monitoring data (called PUSH mode). Nightingale supports a variety of data receiving interfaces, including OpenTSDB, Open-Falcon, PrometheusRemoteWrite, Datadog and other protocols.\nPush Example # Taking the OpenTSDB protocol as an example, the Nightingale interface path is /opentsdb/put, the HTTP Method is POST, and the monitoring data you want to report is placed in the Request Body. The format example is as follows:\n[ { \u0026quot;metric\u0026quot;: \u0026quot;cpu_usage_idle\u0026quot;, \u0026quot;timestamp\u0026quot;: 1637732157, \u0026quot;tags\u0026quot;: { \u0026quot;cpu\u0026quot;: \u0026quot;cpu-total\u0026quot;, \u0026quot;ident\u0026quot;: \u0026quot;c3-ceph01.bj\u0026quot; }, \u0026quot;value\u0026quot;: 30.5 }, { \u0026quot;metric\u0026quot;: \u0026quot;cpu_usage_util\u0026quot;, \u0026quot;timestamp\u0026quot;: 1637732157, \u0026quot;tags\u0026quot;: { \u0026quot;cpu\u0026quot;: \u0026quot;cpu-total\u0026quot;, \u0026quot;ident\u0026quot;: \u0026quot;c3-ceph01.bj\u0026quot; }, \u0026quot;value\u0026quot;: 69.5 } ] Obviously, the outermost layer of JSON is an array. If you only report one piece of monitoring data, you can also omit the outer brackets and directly report the object structure:\n{ \u0026quot;metric\u0026quot;: \u0026quot;cpu_usage_idle\u0026quot;, \u0026quot;timestamp\u0026quot;: 1637732157, \u0026quot;tags\u0026quot;: { \u0026quot;cpu\u0026quot;: \u0026quot;cpu-total\u0026quot;, \u0026quot;ident\u0026quot;: \u0026quot;c3-ceph01.bj\u0026quot; }, \u0026quot;value\u0026quot;: 30.5 } The server will judge whether the reported data is an array or a single object by checking whether the first character is [, and automatically perform the corresponding decoding. If you think that the reported content occupies too much bandwidth, you can also gzip compress the Request Body in your program, and add the Content-Encoding: gzip Header in the HTTP Header.\nYou can Google or ask GPT for the meaning of each field, with the keyword \u0026ldquo;the meaning of each field in OpenTSDB data format\u0026rdquo;. Here I will briefly explain:\n metric: The name of the monitoring indicator, usually an English word, and multiple words are connected with underscores, such as cpu_usage_idle timestamp: Timestamp, in seconds, indicating the collection time of monitoring data tags: Tags, usually a map structure, where the key is the tag name and the value is the tag value, used to describe various dimension information or meta-information of the indicator value: The value of the monitoring data, usually a number, indicating the value of the indicator ð¢ Note the ident tag. ident is the abbreviation of identity, representing the unique identifier of the device. If there is an ident tag in the tags, n9e will consider that this monitoring data comes from a certain machine, and will automatically obtain the value of ident and register it in Nightingale\u0026rsquo;s machine list.\n Other commonly used interface paths:\n /openfalcon/push: Open-Falcon data receiving interface /prometheus/v1/write: Prometheus Remote Write data receiving interface How to Authenticate # If your Nightingale is exposed to the public network, then everyone can push monitoring data to you, which is obviously unsafe. So we suggest:\n Do not expose Nightingale to the public network If you really need to expose it to the public network, use the HTTPS protocol and enable Basic Auth authentication at the same time How to enable Basic Auth authentication? Configure in etc/config.toml:\n[HTTP.APIForAgent] Enable = true [HTTP.APIForAgent.BasicAuth] user001 = \u0026quot;Pa55word01\u0026quot; user002 = \u0026quot;Pa55word02\u0026quot; Your own program calls Nightingale\u0026rsquo;s interface to report monitoring data. At this time, your program is equivalent to an agent role, so it is related to the configuration of HTTP.APIForAgent.\n The Enable under HTTP.APIForAgent must be set to true to enable related interfaces; otherwise, calling those data reporting interfaces will result in a 404 Not Found error. The configuration under HTTP.APIForAgent.BasicAuth is the username and password, in the format of username = \u0026quot;password\u0026quot;. You can set multiple users, and the authentication method for these users is Basic Auth. If there are no users configured under HTTP.APIForAgent.BasicAuth, it means that no authentication is required, and anyone can push data. ð´ Note: There is also a HTTP.APIForService configuration section next to HTTP.APIForAgent, which is used to provide some interfaces for n9e-edge, that is, the edge computer room deployment mode of Nightingale. If you are not using n9e-edge, you must disable HTTP.APIForService, that is, set Enable under HTTP.APIForService to false to avoid security risks.\n "}),e.add({id:38,href:"/docs/faq/global-callback/",title:"Global Callback is Deprecated, How to Handle Global Scenarios?",description:"The new version of Nightingale has removed the global callback function, but some scenarios still require global callbacks. This article introduces how to use subscription rules to replace global callbacks.",content:"It is planned to remove the global callback in the V9 version. If you are currently using the global callback function, you need to migrate to the new method as soon as possible. So, what method can be used to replace the global callback?\nThe answer is: Subscription Rules ( ð click to view usage instructions).\nThe menu entry for subscription rules is: Monitors - Rules - Subscription Rules TAB. Below is the interface for creating a subscription rule:\nWe can create a subscription rule and then subscribe to all alert events, that is, do not add filtering conditions and directly select all levels. In this way, any alert event will match this subscription rule. Note:\n The above image is from an older version, where the data source type still must be selected. Therefore, strictly speaking, the subscription rule in the above example is not for all alert events, but for all alert events of the Prometheus data source type. Future versions will be optimized, and the data source type will be made non-mandatory, meaning you can subscribe to alert events of all data source types. Then, select a notification rule. In this way, all alert events can go through this notification rule. Compared with the previous global callback, there are two typical advantages:\n In subscription rules, you can set some filters, such as subscribing only to alert events of a certain data source type or a certain level. All alert events go through a unified notification rule. The notification rule c
1an have an event processor Pipeline, and different notification media can be used. Different notification media can also have fine-grained filtering configurations, which is overall more flexible than the global callback. "}),e.add({id:39,href:"/docs/faq/cpu-alerting/",title:"Should 'High CPU Load' Trigger an Alert?",description:"Should high CPU load on a server trigger an alert? This question has puzzled many practitioners in operation and maintenance monitoring. This article attempts to provide some suggestions.",content:"Should high CPU load trigger an alert?\n If we don\u0026rsquo;t set an alert, we\u0026rsquo;re afraid of being blamed for missing alerts when problems occur. If we do set an alert, there seems to be too much noise, and engineers automatically ignore them. How awkward\u0026hellip;\nThe adult world is not black and white. To discuss this seriously, we need to add many qualifiers. To avoid ambiguity and align understanding, let me first introduce some prerequisite knowledge (principles).\nPrerequisite Knowledge (Principles) # Alerts should have different urgency levels. Some companies even stipulate 6 levels (probably their own engineers can\u0026rsquo;t sort them out clearly\u0026hellip;). Usually, 3 levels are sufficient:\n Critical: Business has been affected and requires immediate handling. Usually, multiple highly intrusive notification channels are used to send alert messages together, such as phone calls + SMS + IM + email. For example, a significant drop in e-commerce order volume is an emergency alert. Warning: No immediate handling is needed. A work order can be automatically created for gradual processing. However, it must be handled; otherwise, it may lead to major failures. Alerts are usually sent, but with less intrusive notification channels. For example, the disk usage of an important machine has reached 95% and may be full in another 24 hours; or a domain certificate will expire in 3 days. Info: Only generates an alert event without sending a notification. It is equivalent to extracting some slightly important information from massive metrics. If a failure occurs, this information serves as clues for troubleshooting. For example, a Pod is evicted and migrated; or a user has too many failed login attempts. Overall, they can be divided into two categories:\n Need to be handled: Critical, Warning No need to be handled: Info Among them, Info is not critical and can be configured or not. It can definitely wait until your monitoring and fault location system is more refined. We focus on the first two levels: Critical and Warning. These two levels have one thing in common: they both NEED TO BE HANDLED! In the English world, this is usually called \u0026ldquo;actionable\u0026rdquo; (which feels very accurate).\nSo, should we configure alerts for high CPU load?\nLogic for Formulating CPU Alerts # If there are follow-up actions after a CPU alert is triggered, it should be configured. Even if the action is logging into the machine to take a look and writing a few follow-up conclusions, it counts as an action. If there are no follow-up actions, there\u0026rsquo;s no need to configure it. For example, if you see the alert, get used to it, and directly ignore it, that\u0026rsquo;s not an action, and the alert shouldn\u0026rsquo;t be configured. Alternatively, it can be configured as an Info level, only generating an alert event without sending a notification. In fact, this logic applies to all alert rule configurations. All alert rules should be actionable. Therefore, in theory, each alert rule should correspond to an SOP (Standard Operating Procedure). Both Prometheus and Nightingale\u0026rsquo;s alert rules have an Annotations field. Typical fields that should be placed in Annotations are SOP URL and Dashboard URL.\nMany people reading this may think that this requires a lot of work. Each alert rule needs to have an SOP sorted out (SOPs usually differ between companies; some SOPs for middleware and databases may be the same). Previously, they just found some alert rules online, imported them, and thought that was it, not expecting there to be more work!\nIn fact, compared to building a monitoring system, this is the more valuable part!\n"}),e.add({id:40,href:"/docs/faq/biz-team-subscribe-infra-alarms/",title:"Should upper-layer businesses receive alarms from underlying infrastru
1cture?",description:"I am a DEV or SRE of a business application. My application depends on underlying services and infrastructure, such as basic networks, Kubernetes, MySQL, and cash register services. Should I receive alarms if these basic services have problems? There is a subscription rule in Nightingale. Is it designed for this purpose?",content:"A friend asked: I am a DEV or SRE of a business application. My application depends on underlying services and infrastructure, such as basic networks, Kubernetes, MySQL, and cash register services. Should I receive alarms if these basic services have problems? There is a subscription rule in Nightingale. Is it designed for this purpose?\nThis article presents the author\u0026rsquo;s personal understanding, for reference only.\nFirst, please read the previous article \u0026ldquo;Should high CPU load trigger an alarm?\u0026rdquo;, which mentions a point: Only actionable alarm rules are meaningful!\nSo, it depends on your situation:\n1. If your service is deployed in a single computer room, and you can do nothing about problems with these infrastructure and services but passively wait for recovery (i.e., you have no SOP), then receiving these alarms is of little significance.\nThe recommended approach at this time is: Create a visualization page, put the key SLI of the infrastructure and services you depend on, so that you can understand the health status of each infrastructure and service by viewing the UI trend chart. Facebook has an internal product called SLICK, which follows a similar logic. Our startup\u0026rsquo;s Flashcat has a feature called \u0026ldquo;Fire Fighting Chart\u0026rdquo; with a similar effect. This is a common practice.\n2. If you have an SOP, such as the ability to switch traffic, then subscribing to such alarms is meaningful.\nHowever, you may not understand the key indicators of many underlying services, so it is better to have a specification. For example, when configuring alarm rules, the person in charge of each underlying service, if they think a certain alarm rule is very important and the corresponding alarm will affect upper-layer services, can mark that rule with a special label. For example, advertise=mysql indicates all MySQL-related alarms that need to be known, and advertise=k8s indicates all Kubernetes-related alarms that need to be known. Then, DEVs and SREs of upper-layer applications can subscribe to such labels to be informed of relevant alarms.\nIn addition\nFor services like MySQL and cash registers, a better way than subscribing to their SLI alarms is to bury points in upper-layer applications yourself, mainly collecting the number of requests, failures, and delays.\nBecause from MySQL\u0026rsquo;s perspective, its SLI indicators are for the entire instance, while if you bury points in upper-layer applications, the indicators can be refined to be related to this service, or even to specific business scenarios of this service, making it more precise.\n"}),e.add({id:41,href:"/docs/faq/how-to-import-prometheus-rules/",title:"How to import Prometheus rules?",description:"How to import Prometheus alert rules into Nightingale?",content:"Many people in the Prometheus ecosystem have shared alerting rules, such as this project:\n https://github.com/samber/awesome-prometheus-alerts/tree/master/dist/rules Each directory contains alerting rules in YAML format. For example, the host-and-hardware directory includes common alerting rules for node-exporter. Want to directly import these rules into Nightingale? Please refer to the following steps.\nVersion # Please use Nightingale version 8.2.0 or later.\nImport Steps # As shown in the screenshot above, go to the alerting rules page and select import to import Prometheus-format alerting rules. Note that the YAML-formatted rule content starts with groups, which contains multiple group entries. Each group has a name and rules, where rules is an array of specific alerting rules. Nightingale will ignore the group\u0026rsquo;s name during processing and directly import the content in rules.\nAfter the import is complete, you usually need to associate notification rules to enable alert notifications. The method is: select the alert
1ing rules in batches, then click \u0026ldquo;More Operations\u0026rdquo; in the upper right corner to batch update the alerting rules:\nIn the batch update pop-up, select \u0026ldquo;Notification rule\u0026rdquo; as the field, then choose the corresponding notification rule and click \u0026ldquo;Confirm\u0026rdquo;. The screenshot is as follows:\n"}),e.add({id:42,href:"/docs/faq/",title:"FAQ",description:"",content:""}),e.add({id:43,href:"/docs/usecase/",title:"Use Case",description:"",content:""}),e.add({id:44,href:"/docs/practice/",title:"Practice",description:"",content:""}),e.add({id:45,href:"/docs/usage/",title:"Quick Start",description:"Nightingale usage book",content:""}),e.add({id:46,href:"/docs/agent/",title:"Collector",description:"Categraf, Telegraf, Grafana-Agent, Datadog-Agent",content:""}),e.add({id:47,href:"/docs/install/",title:"Installation",description:"How to install nightingale project",content:""}),e.add({id:48,href:"/docs/prologue/",title:"Prologue",description:"Nightingale's introduction",content:""}),e.add({id:49,href:"/docs/",title:"Nightingale",description:"Nightingale is a cloud native monitoring system",content:""}),search.addEventListener("input",t,!0);function t(){const s=5;var n=this.value,o=e.search(n,{limit:s,enrich:!0});const t=new Map;for(const e of o.flatMap(e=>e.result)){if(t.has(e.doc.href))continue;t.set(e.doc.href,e.doc)}if(suggestions.innerHTML="",suggestions.classList.remove("d-none"),t.size===0&&n){const e=document.createElement("div");e.innerHTML=`No results for "<strong>${n}</strong>"`,e.classList.add("suggestion__no-results"),suggestions.appendChild(e);return}for(const[r,a]of t){const n=document.createElement("div");suggestions.appendChild(n);const e=document.createElement("a");e.href=r,n.appendChild(e);const o=document.createElement("span");o.textContent=a.title,o.classList.add("suggestion__title"),e.appendChild(o);const i=document.createElement("span");if(i.textContent=a.description,i.classList.add("suggestion__description"),e.appendChild(i),suggestions.appendChild(n),suggestions.childElementCount==s)break}}})()
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.