aiesimulator hangs on MNIST example

[jbell-astrobotic]

I'm able to compile things and aiesimulator is running, but the output eventually stalls at the end waiting for core(s) of graph aie_dut to finish execution. I set a simulation cycle timeout so it eventually completes but the outputs are empty.

Without the timeout the output will be like this indefinitely:

Enabling core(s) of graph aie_dut
1 us Waiting for core(s) of graph aie_dut to finish execution ...
7 us 

Here are the full logs:

make -C aie/mnist sim
make[1]: Entering directory '/home/joseph.bell/projects/Vitis-Tutorials/AI_Engine_Development/AIE-ML/Design_Tutorials/09-VEK280-MNIST-End2End/aie/mnist'
aiesimulator --pkg-dir=./Work --input-dir=./ --output-dir=./aiesimulator_output --display-run-interval=1000 --simulation-cycle-timeout 20000
                                           AI Engine Simulator
                                     Version 2025.2 (linux64-bit)
                                     SW Build 6299465 on 2025-11-14-04:55:54
                          Copyright 1986-2022 Xilinx, Inc. All Rights Reserved.
                          Copyright 2022-2025 Advanced Micro Devices, Inc. All Rights Reserved.
AIEMLsim feature license is found.
INFO: Executing config: ./Work/config/scsim_config.json
set use_simple_noc to: 1
Loading device config from: /home/joseph.bell/vitis/2025.2/Vitis/aietools/data/aie_ml/devices/VC2802.json
[INFO] AIE Frequency: 1.25e+09 Hz 
Removed previously generated option file
AIE_WORK_DIR = /home/joseph.bell/projects/Vitis-Tutorials/AI_Engine_Development/AIE-ML/Design_Tutorials/09-VEK280-MNIST-End2End/aie/mnist/./Work
Initializing AIE driver...
Initializing ADF API...
XAIEFAL: INFO: Resource group Avail is created.
XAIEFAL: INFO: Resource group Static is created.
XAIEFAL: INFO: Resource group Generic is created.
IP-INFO: [ps_i30_ps_main] IP loaded.
Warning: ps_lib_path is empty, And supported simulators are XSIM, RIVIERA, XCELIUM, VCS and QUESTA

Info: DEVICE FILE: /home/joseph.bell/vitis/2025.2/Vitis/aietools/data/aie_ml/devices/VC2802.json

Info: AIE SOLUTION FILE: ./Work/arch/aieshim_solution.aiesol
 AIE CLUSTER LIB PATH : libaie2_cluster_msm_v1_0_0.osci.so
[INFO]: Disable Unused Core Tiles
xpe directory: /home/joseph.bell/projects/Vitis-Tutorials/AI_Engine_Development/AIE-ML/Design_Tutorials/09-VEK280-MNIST-End2End/aie/mnist/./Work
Found xpe file /home/joseph.bell/projects/Vitis-Tutorials/AI_Engine_Development/AIE-ML/Design_Tutorials/09-VEK280-MNIST-End2End/aie/mnist/./Work/reports/mnist_app.xpe
[AIESIM_OPTIONS]: aiesim_options file path ./aiesimulator_output/aiesim_options.txt
[INFO]: AIE Cycle-approximate Model
INFO: Running AIE2 MTMODEL Simulation with 4 threads
[INFO]: Disable Unused Memory Tiles 
AIE2 ISS r1p8
ISS disables unused tiles
Iss used row and col 0 0
Iss used row and col 0 1
Iss used row and col 0 2
Iss used row and col 0 3
Iss used row and col 0 4
Iss used row and col 0 5
Iss used row and col 0 6
Iss used row and col 0 7
Iss used row and col 0 8
Iss used row and col 0 9
Iss used row and col 0 10
Iss used row and col 0 11
Iss used row and col 0 12
Iss used row and col 0 13
Iss used row and col 0 14
Iss used row and col 0 15
Iss used row and col 0 16
Iss used row and col 0 17
Iss used row and col 0 18
Iss used row and col 0 19
Iss used row and col 0 20
Iss used row and col 0 21
Iss used row and col 0 22
Iss used row and col 0 23
Iss used row and col 0 24
Iss used row and col 0 25
Iss used row and col 0 26
Iss used row and col 1 18
Iss used row and col 1 19
Iss used row and col 1 20
Iss used row and col 1 21
Iss used row and col 1 22
Iss used row and col 1 23
Iss used row and col 1 24
Iss used row and col 1 25
Iss used row and col 1 26
[INFO]: Array constructed 
[INFO]: Shim row constructed 
[INFO]: Mem row constructed 
--------------------------------------------------------------------------------------------------
SLAVE STREAM INFO:  | Port Name         | Port Width        | Stream Type       | Channel Index     
                    | S00_AXIS          | 64                | PL STREAM         | 145               
                    | S01_AXIS          | 64                | PL STREAM         | 148               
                    | S02_AXIS          | 64                | PL STREAM         | 160               
                    | S03_AXIS          | 64                | PL STREAM         | 176               
                    | S04_AXIS          | 64                | PL STREAM         | 184               
                    | S05_AXIS          | 64                | PL STREAM         | 192               
                    | S06_AXIS          | 64                | PL STREAM         | 200               
                    | S07_AXIS          | 64                | PL STREAM         | 208               
--------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------
MASTER STREAM INFO: | Port Name         | Port Width        | Stream Type       | Channel Index     
                    | M00_AXIS          | 64                | PL STREAM         | 156               
--------------------------------------------------------------------------------------------------
Using new arbitration logic
Enabled fast PM writes.
Enabled fast DM writes.
INFO: AIESimulator app running for 20000 cycles
0 s IP-INFO: Starting to send data from file: .//data/ifm_i.txt
IP-INFO: AMD recommends using the CSV based file format for simulating PLIO based input and outputs which provides support for enhanced data control. For more details, please refer UG 1076 https://docs.xilinx.com/r/en-US/ug1076-ai-engine-environment/Simulation-Input-and-Output-Data-Streams 
IP-INFO: Starting to send data from file: .//data/wts1_i.txt
IP-INFO: Starting to send data from file: .//data/wts3_i.txt
IP-INFO: Starting to send data from file: .//data/wts5_0_i.txt
IP-INFO: Starting to send data from file: .//data/wts5_1_i.txt
IP-INFO: Starting to send data from file: .//data/wts5_2_i.txt
IP-INFO: Starting to send data from file: .//data/wts5_3_i.txt
IP-INFO: Starting to send data from file: .//data/wts7_i.txt
Starting to receive data into file: ./aiesimulator_output/data/ofm_o.txt
IP-INFO: [ps_i30_ps_main] IP started.
Loading elfs of graph aie_dut...
Initializing graph aie_dut...
Resetting cores of graph aie_dut...
Configuring DMAs of graph aie_dut...
Configuring PL-Interface for graph aie_dut...
Set 1 iterations for the core(s) of graph aie_dut
Enabling core(s) of graph aie_dut
1 us Waiting for core(s) of graph aie_dut to finish execution ...
15 us Exiting!
Cores are done executing but the simulation will run for some more cycles to allow PLIO to be flushed
Stopping Simulator.

Info: /OSCI/SystemC: Simulation stopped by user.
--------------------------------------------------------------------------------
| Intf Type   | Port Name                          | Type  | Throughput(MBps)  |
--------------------------------------------------------------------------------
| plio        | PLIO_i                             | IN    | 5006.20           |
|             | PLIO_wts1                          | IN    | 5128.21           |
|             | PLIO_wts3                          | IN    | 3363.78           |
|             | PLIO_wts5_0                        | IN    | 3348.57           |
|             | PLIO_wts5_1                        | IN    | 3348.57           |
|             | PLIO_wts5_2                        | IN    | 3348.57           |
|             | PLIO_wts5_3                        | IN    | 3348.57           |
|             | PLIO_wts7                          | IN    | 3357.78           |
|             | PLIO_o                             | OUT   | 0.00              |
./aiesimulator_output/data/ofm_o.txt
IP-INFO: deleting ip PSIP_ps_i30 
IP-INFO: deleting packet ip 
IP-INFO: deleting packet ip 
IP-INFO: deleting packet ip 
IP-INFO: deleting packet ip 
IP-INFO: deleting packet ip 
IP-INFO: deleting packet ip 
IP-INFO: deleting packet ip 
IP-INFO: deleting packet ip 
IP-INFO: deleting packet ip 
[INFO] : Simulation Finished, Sim result: 0 Total Simulation time 16 us, Wall clock time 63.62 s 
AIEMLsim feature license is released.
make[1]: Leaving directory '/home/joseph.bell/projects/Vitis-Tutorials/AI_Engine_Development/AIE-ML/Design_Tutorials/09-VEK280-MNIST-End2End/aie/mnist'

The x86 simulation works just fine.
I'm on Ubuntu24.

[jbell-astrobotic]

Some additional logs that get generated after a hang - this prints out continuously

Sending SIGINT to 36 uicores.
Sending SIGINT to 36 uicores.
Sending SIGINT to 36 uicores.
Sending SIGINT to 36 uicores.
Sending SIGINT to 36 uicores.
Sending SIGINT to 36 uicores.
Sending SIGINT to 36 uicores.
Sending SIGINT to 36 uicores.
Sending SIGINT to 36 uicores.
Sending SIGINT to 36 uicores.
Sending SIGINT to 36 uicores.
Sending SIGINT to 36 uicores.
Sending SIGINT to 36 uicores.
Sending SIGINT to 36 uicores.
Sending SIGINT to 36 uicores.
Sending SIGINT to 36 uicores.
Sending SIGINT to 36 uicores.
Sending SIGINT to 36 uicores.
Sending SIGINT to 36 uicores.
Sending SIGINT to 36 uicores.
Sending SIGINT to 36 uicores.
Sending SIGINT to 36 uicores.