Introduction

Since many days, I had problem with web response time and increase time after time, and be in timeout when periodics running the morning (4h ~ 9h).

The SSH terminal lag too.

Resources are ok

I have checked many points of control:

• CPU OK (less than 20 %)
• memory RAM OK (8Go free)
• Swap 100% free
• zpool (ZFS) at 40% filled
• Load average < 0.1

So, crazy problem, why this lag :(

Commands

After many days, I tried many commands, and one has been interessing and give me something in the road of the resolution of this problem:

systat

and give me:

                    /0   /1   /2   /3   /4   /5   /6   /7   /8   /9   /10
     Load Average   

                    /0%  /10  /20  /30  /40  /50  /60  /70  /80  /90  /100
root           idle XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
root           idle XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
root           idle XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
root           idle XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
root           idle XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
root           idle XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
root           idle XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
root           idle XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
root           idle XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
root           idle XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
root           idle XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
root           idle XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
root        vnlru  XXX

vnlru

But, what is this vnlru???

Simple, need read the documentation: (freebsd handbook - chapter 10.20)[https://docs.freebsd.org/en/books/faq/].

A vnodes problems, interessing

So, if I understand right, the vnodes are full.

vnodes

Now, I check the vnodes.

This is the commands to find the vnodes:

root@myserver:/root # sysctl -n vfs.numvnodes
500000
root@myserver:/root # sysctl -n kern.maxvnodes
500000

Oh! So yes, there is a bug problem.

I found a script doing the calculation for me: (https://www.unix-experience.fr/monitoring/freebsd_vnodes/)[https://www.unix-experience.fr/monitoring/freebsd_vnodes/]

So we can increase the value kern.maxvnodes, but need increase little steps by steps, and check the RAM because it will increase it.

The command to increase the vnodes:

sysctl kern.maxvnodes=600000

To have it permanent (keep it after reboot), put in the file /etc/sysctl.conf:

kern.maxvnodes=600000

Zabbix monitoring

Now, I will add this points of control in the monitoring tool: Zabbix.

On the server, I add a new file /usr/local/etc/zabbix44/zabbix_agentd.conf.d/sysctl.conf and put it inside:

UserParameter=sysctl.kern.maxvnodes,sysctl -n kern.maxvnodes
UserParameter=sysctl.vfs.numvnodes,sysctl -n vfs.numvnodes

I restart the Zabbix agent service.

Then in Zabbix, I create a new template FreeBSD sysctl values, and create a new application FreeBSD sysctl.

I create 2 new items in this application with the key:

• sysctl.kern.maxvnodes
• sysctl.vfs.numvnodes

TO finish, I apply this template to all my FreeBSD hosts.

I add nice panels in my Grafana tool (dashboards) ;)

PS: if we want Zabbix alerts, we need create a trigger in theses values.