the console is not accessed

Hi Nacho,

Thanks for reaching out.

Brief explanation about the cause of the issue:

The above error will occur when the /var/log directly is full and is under 10 % of memory.
We need to check the reason for the var/log directory to reach its saturation limit.
/var/log directory gets filled up due to various reasons like agent crash logs, monitoring system logs which get stored here and older logs will be deleted but in this case, for some reason, it did not clear itself and the memory usage went below 10% that might have caused this issue.

Could you please try and answer the below queries:

1)Can we access the switch via eAPI from the browser?(Management API needs to be enabled in the switch configuration).

2)Did you try to scp from your Local Machine to the device?

3)Did you try  to access the switch to retrieve the logs by scp from the jump server(if one is present)?

Could you please try the above steps and let us know if you are able to access the switch.

Incase if we run out of all the above steps ,since we are not even able to access the switch via console , reloading is the only solution to clear the var/log directory to free up space.
Var/log directory logs will be deleted after the reload .

If you would like to get RCA regarding why the /var/log directory is getting filled up then feel free to open a case with Arista TAC by dropping a mail to support@arista.com for further investigation by collecting tech-support and history tech-support logs.

Thanks,

Bhavana.

Hi Nacho,

In addition to Bhavna's suggestion, if you are able to access the switch from the front panel port or ssh the management port, you would like to take a backup of the /var/logs, in case you want to open a TAC case to know the RCA. 

Once you have taken the backup, you may truncate/rm the filesystem to free the space. Please find below the sample from my lab switch. I took a backup of /var/log to the folder named test on my desktop as below:

switch#bash

Arista Networks EOS shell

[admin@switch ~]$ df -h

Filesystem Size Used Avail Use% Mounted on

/dev/mmcblk0p1 3.3G 2.9G 320M 91% /mnt/flash

none 1.2G 83M 1.1G 7% /

none 1.2G 83M 1.1G 7% /.overlay

/dev/loop0 908M 908M 0 100% /rootfs-i386

devtmpfs 3.8G 0 3.8G 0% /dev

tmpfs 3.9G 0 3.9G 0% /dev/shm

tmpfs 3.9G 2.0M 3.9G 1% /run

tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup

tmpfs 792M 15M 778M 2% /var/core

tmpfs 64M 0 64M 0% /.deltas

tmpfs 2.0G 24M 2.0G 2% /var/shmem

tmpfs 792M 189M 604M 24% /var/log  -------------------------> This is the concerned filesystem in your case 

tmpfs 3.9G 0 3.9G 0% /run/netns

tmpfs 396M 232K 396M 1% /var/tmp

tmpfs 1.2G 872K 1.2G 1% /tmp

tmpfs 5.8G 9.3M 5.8G 1% /var/shmem/stashes

[admin@switch ~]$

[admin@switch ~]$ cd /var/log/

[admin@switch log]$ scp * Username@IP_address:/Users/user/Desktop/test  -------------------------> take backup

Warning: Permanently added '10.95.68.212' (ECDSA) to the list of known hosts.

Password:

- Once you have taken the back, truncate or rm all the directories, occupying large space. I am doing it for the agent logs. 

#bash

Arista Networks EOS shell

[admin@switch ~]$ cd /var/log/agents 

[admin@switch ~]$sudo truncate --size=0 * 

Or delete the files as below:

[admin@switch ~]$sudo rm *

[admin@switch ~]$exit