Posted on May 3, 2020 1:44 pm
 |  Asked by Todd Riemenschneider
 |  257 views
RESOLVED
0
0
Print Friendly, PDF & Email

Hi I am running into an issue with the latest lab image. I build the lab using the documentation posted with the ceoslab images. I get to the point where I set up eth1 so the two devices can ping each other. Then I try and enable routing between two devices,  ospf in this particular instance and it will say ospf inactive.  I will then get “(No space left on device)”.  It’s really basic config stuff, the same configs work fine with and earlier  ceos image 4.20.5F. I am aware of the change between ceossim and ceoslab.  I tried to increase the resources in docker to allocate 3 Gigs per image just to see if it would help but to no avail. So I changed it back figuring it may be internal to the image. I really do not know docker that well and could be it be docker related? My docker is running the latest updates.

Internal build version: 4.23.3M-16433654.4233M

Internal build ID:      ae88d3a4-9741-46c0-9f25-81da7e252b4b

Here’s the error:

ceos1#sh run 

! Command: show running-config

% Error displaying system:/running-config (No space left on device)

ceos1# sh ver

 cEOSLab

Hardware version:      

Serial number:         

Hardware MAC address:  0242.ac22.bd71

System MAC address:    0242.ac22.bd71

Software image version: 4.23.3M

Architecture:           x86_64

Internal build version: 4.23.3M-16433654.4233M

Internal build ID:      ae88d3a4-9741-46c0-9f25-81da7e252b4b

cEOS tools version: 1.1

Uptime:                 0 weeks, 0 days, 11 hours and 27 minutes

Total memory:           2037620 kB

Free memory:            365512 kB

ceos1#

 

Configs prior to the error:

ceos2#sh run 

! Command: show running-config

! device: ceos2 (cEOSLab, EOS-4.23.3M)

!

transceiver qsfp default-mode 4x10G

!

agent Bfd shutdown

agent PowerManager shutdown

agent LedPolicy shutdown

agent Thermostat shutdown

agent PowerFuse shutdown

agent StandbyCpld shutdown

agent LicenseManager shutdown

!

hostname ceos2

!

spanning-tree mode mstp

!

no aaa root

!

username arista privilege 15 secret sha512 $6$uaTXm2.QUySurtn.$4bWcRhjWXbQ2RIeNTTVCWRWSi82ULjfFn8ZDiIKxYolvIn.zgKRzPch2/P5noPDCUPlqBertROuCluXEKyZFP/

!

interface Ethernet1

   no switchport

   ip address 10.0.0.2/24

!

interface Ethernet2

!

interface Loopback0

   ip address 2.2.2.2/32

!

ip routing

!

router ospf 1

   network 10.0.0.0/24 area 0.0.0.0

   max-lsa 12000

!

management api http-commands

   no shutdown

!

end

ceos2# show ip ospf nei

! OSPF inactive

ceos2#sh run 

! Command: show running-config

% Error displaying system:/running-config (No space left on device)

ceos2#

ceos1>show int status

Port       Name   Status       Vlan     Duplex Speed  Type            Flags Encapsulation

Et1               connected    routed   full   unconf EbraTestPhyPort                    

Et2               connected    1        full   unconf EbraTestPhyPort                    

ceos1>

I reboot and tried the same thing only this time using BGP and got the same error.

This was my build procedure:

docker import /Users/toddriemenschneider/Downloads/cEOS64-lab-4.23.3M.tar.xz ceosimage:4.23.3M

# create docker instances with needed environment variables

docker create –name=ceos1 –privileged -p 443:443 -p 2201:22 -p 2301:23 -e INTFTYPE=eth -e ETBA=1 -e SKIP_ZEROTOUCH_BARRIER_IN_SYSDBINIT=1 -e CEOS=1 -e EOS_PLATFORM=ceoslab -e container=docker -i -t ceosimage:4.23.3M /sbin/init systemd.setenv=INTFTYPE=eth systemd.setenv=ETBA=1 systemd.setenv=SKIP_ZEROTOUCH_BARRIER_IN_SYSDBINIT=1 systemd.setenv=CEOS=1 systemd.setenv=EOS_PLATFORM=ceoslab systemd.setenv=container=docker

docker create –name=ceos2 –privileged -p 8443:443 -p 2202:22 -p 2302:23 -e INTFTYPE=eth -e ETBA=1 -e SKIP_ZEROTOUCH_BARRIER_IN_SYSDBINIT=1 -e CEOS=1 -e EOS_PLATFORM=ceoslab -e container=docker -i -t ceosimage:4.23.3M /sbin/init systemd.setenv=INTFTYPE=eth systemd.setenv=ETBA=1 systemd.setenv=SKIP_ZEROTOUCH_BARRIER_IN_SYSDBINIT=1 systemd.setenv=CEOS=1 systemd.setenv=EOS_PLATFORM=ceoslab systemd.setenv=container=docker

# create docker networks

docker network create net1

docker network create net2

# connect docker instances to the networks

docker network connect net1 ceos1

docker network connect net1 ceos2

docker network connect net2 ceos1

docker network connect net2 ceos2

# start the instances

docker start ceos1

docker start ceos2

The image boots fine I can get in and enable management features like eapi its when I want to turn on routing things go south.

 

I’ll attach some docker info concerning these containers. Don’t really know if it will help.

Thanks for any advice.

~Todd

 

0
Posted by Tamas Plugor
Answered on May 3, 2020 2:00 pm

Hi Todd,

Does this happen with any newly created docker images? If yes, you might have run out of space in "/var/lib" on the host.
If this only happens with cEOS 4.23.3M, did you check what's occupying the free space?
You could run 'df -h' in bash to check if your partitions are full, then if you find which one of them is full, you could list the files to see if an agent is crashing in "/var/log/agents" or if just the flash is full

HTH,
Tamas

0
Answered on May 3, 2020 2:36 pm

Tamas,
Thanks for reaching out. To answer your question the only image I have been successfully able to run routing with, is the old ceossim image 4.20.5F. Every other image that is part of the ceoslab build, routing has not worked for me. Specifically !ospf not active is the common error I receive among all the newer images. I have tried a few different images over the past year and I will say that this is the only time I remember running out of disk space though. So that's why I'm posting.

I've attached the command outputs

Attachments:
The Rib agent seems to be crashing, can you share the output of one of those Rib files?
(Tamas Plugor at May 3, 2020 2:53 pm)
Within the Rib files I see a common error at the same point in the file. Bare in mind I do not know what I'm looking at other than a common trait. Exception thrown is Tac::Exception("Invalid IntfId ")
(Todd Riemenschneider at May 3, 2020 2:59 pm)
I uploaded the file. Let me know if it didn't come through. Thanks.
(Todd Riemenschneider at May 3, 2020 4:00 pm)
0
Answered on May 3, 2020 3:06 pm

Here you go.

Attachments:
The problem is that your interface name is invalid: "Invalid IntfId " there's an extra 'h' at the end, it should be "Ethernet1", although not sure where that typo comes from for you, do you load the config during boot? I did a quick test on my CentOS VM but it works fine for me, the only difference is that I use slightly different env vars, but should work with yours too:
docker create --name=ceos0504 -p 4000:8080 -p 4001:6030 -p 4002:6061 -p 4003:8081 -p 4423:443 -p 2223:22 --privileged -e INTFTYPE=eth -e ETBA=4 -e SKIP_ZEROTOUCH_BARRIER_IN_SYSDBINIT=1 -e CEOS=1 -e EOS_PLATFORM=ceoslab -e container=docker -i -t ceosimage:cEOS-lab-4.23.3M-64 /sbin/init systemd.setenv=INTFTYPE=eth systemd.setenv=ETBA=4 systemd.setenv=SKIP_ZEROTOUCH_BARRIER_IN_SYSDBINIT=1 systemd.setenv=CEOS=1 systemd.setenv=EOS_PLATFORM=ceoslab systemd.setenv=container=docker systemd.setenv=MAPETH0=1 systemd.setenv=MGMT_INTF=eth0
(Tamas Plugor at May 4, 2020 9:31 pm)
So I load the config from pyeapi once the device is online and I’ve enabled http/https management interface. The script works fine with the older platform ceos-sim. So I use the same script. All the configs load properly. It just will say !OSPF inactive then it will run out of space. When you say there’s an extra ”h” I’m not sure where that would be either? configs load as I would expect. It’s weird.
(Todd Riemenschneider at May 5, 2020 3:57 pm)
For sanity I have also simply done it from the cli, and it gives me the same result. I didn’t know if there was a feature I needed to enable in the new versions of eos. Then it started running out of space and I figured it was something else.
(Todd Riemenschneider at May 5, 2020 4:03 pm)
Hey Tamas, thanks for checking into this. I do appreciate it. I tried building the ceos using your build script and it loaded no issues. However I no longer could connect remotely. Tried with with pyeapi. I updated the eapi.conf script with new ports prior to trying. Tried verbose ssh on the new port and kept getting ssh_exchange_identification connection closed by remote host. So I logged in using 'docker exec -it ceos1 Cli', tried enabling ospf got the same !ospf inactive then ran out of disk space. I'm doing this on a MacBook Pro with 8Gig and 128G of storage so I have to be mindful of the apps I'm using trying to conserve what space I have left. At this point beginning to question whether its worth it with all the remote labs out there now. I do appreciate your efforts. If it was a simple as the ceos-sim I would continue updating but somethings changed and my laptop is making this way too difficult. Thanks again.
(Todd Riemenschneider at May 6, 2020 4:03 pm)
Hi Todd, I tried to reach you out via email but not sure if you’ve gotten it, we could do a remote session to check out your setup if that would help! Thanks, Tamas
(Tamas Plugor at May 6, 2020 10:54 pm)
Yea I got your email. Let me know when. -Todd
(Todd Riemenschneider at May 6, 2020 11:56 pm)
0
Posted by Tamas Plugor
Answered on May 11, 2020 4:39 pm

Just in case others hit the same this is because in cEOS the kernel interfaces are using the naming convention of ethX instead of etX and when the interfaces are converted they become EthernethX with an extra 'h' (tracked under bug397410). This only affects ospf and is-is and only in cEOS, which will be fixed in an upcoming release. One workaround is to use port-channel interface

Thanks,
Tamas

Thanks for the assistance Tamas!
(Todd Riemenschneider at May 12, 2020 6:07 pm)

Post your Answer

You must be logged in to post an answer.