Posted on February 22, 2021 11:59 am
 |  Asked by Nacho Marín
 |  41 views
0
0
Print Friendly, PDF & Email

Hello,
I cannot access the switch through the console, the error indicates
Login: write lastlog failed: no space left on device
Warning: the followingfilesystems haveless than 10% free space left:
tmpfs (on /var/log) 0%
Is reboot the only solution?
Thanks a lot.

0
Answered on February 23, 2021 3:37 am

Hi Nacho,

Thanks for reaching out.

Brief explanation about the cause of the issue:

The above error will occur when the /var/log directly is full and is under 10 % of memory.
We need to check the reason for the var/log directory to reach its saturation limit.
/var/log directory gets filled up due to various reasons like agent crash logs, monitoring system logs which get stored here and older logs will be deleted but in this case, for some reason, it did not clear itself and the memory usage went below 10% that might have caused this issue.

Could you please try and answer the below queries:

1)Can we access the switch via eAPI from the browser?(Management API needs to be enabled in the switch configuration).

2)Did you try to scp from your Local Machine to the device?

3)Did you try  to access the switch to retrieve the logs by scp from the jump server(if one is present)?

Could you please try the above steps and let us know if you are able to access the switch.

Incase if we run out of all the above steps ,since we are not even able to access the switch via console , reloading is the only solution to clear the var/log directory to free up space.
Var/log directory logs will be deleted after the reload .

If you would like to get RCA regarding why the /var/log directory is getting filled up then feel free to open a case with Arista TAC by dropping a mail to support@arista.com for further investigation by collecting tech-support and history tech-support logs.

 

Thanks,

Bhavana.

0
Posted by Naveen Chandra
Answered on February 23, 2021 8:12 am

Hi Nacho,

In addition to Bhavna's suggestion, if you are able to access the switch from the front panel port or ssh the management port, you would like to take a backup of the /var/logs, in case you want to open a TAC case to know the RCA. 

Once you have taken the backup, you may truncate/rm the filesystem to free the space. Please find below the sample from my lab switch. I took a backup of /var/log to the folder named test on my desktop as below:

 

switch#bash

Arista Networks EOS shell

[admin@switch ~]$ df -h

Filesystem Size Used Avail Use% Mounted on

/dev/mmcblk0p1 3.3G 2.9G 320M 91% /mnt/flash

none 1.2G 83M 1.1G 7% /

none 1.2G 83M 1.1G 7% /.overlay

/dev/loop0 908M 908M 0 100% /rootfs-i386

devtmpfs 3.8G 0 3.8G 0% /dev

tmpfs 3.9G 0 3.9G 0% /dev/shm

tmpfs 3.9G 2.0M 3.9G 1% /run

tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup

tmpfs 792M 15M 778M 2% /var/core

tmpfs 64M 0 64M 0% /.deltas

tmpfs 2.0G 24M 2.0G 2% /var/shmem

tmpfs 792M 189M 604M 24% /var/log  -------------------------> This is the concerned filesystem in your case 

tmpfs 3.9G 0 3.9G 0% /run/netns

tmpfs 396M 232K 396M 1% /var/tmp

tmpfs 1.2G 872K 1.2G 1% /tmp

tmpfs 5.8G 9.3M 5.8G 1% /var/shmem/stashes

[admin@switch ~]$

[admin@switch ~]$ cd /var/log/

 

[admin@switch log]$ scp * Username@IP_address:/Users/user/Desktop/test  -------------------------> take backup

Warning: Permanently added '10.95.68.212' (ECDSA) to the list of known hosts.

Password:

 

- Once you have taken the back, truncate or rm all the directories, occupying large space. I am doing it for the agent logs. 

 

#bash

Arista Networks EOS shell

[admin@switch ~]$ cd /var/log/agents 

[admin@switch ~]$sudo truncate --size=0 * 

 

Or delete the files as below:

 

[admin@switch ~]$sudo rm *

[admin@switch ~]$exit

 

 

 

Post your Answer

You must be logged in to post an answer.