Reproducibility: Code does not match paper's experimental methodology (Table 2, data splits, normalization)

[andermannfelix-max]

Hello,

Thank you for publishing the KALFormer paper and releasing code. After a careful review of the repository, I found several significant discrepancies between the paper's described methodology and the actual implementation, making reproduction of the reported results impossible with the current codebase:

1. No Train/Validation/Test Split (Section 4.1)
   The paper states: "data were chronologically divided into training, validation, and test sets in an 8:1:1 ratio." However, no code in the repository implements any data splitting. All training scripts (kalformer.py, train_ett.py, train_electricity.py, etc.) create a single DataLoader over the entire dataset and train on all data. There is no validation loop and no test evaluation — only training metrics are reported.

2. Hyperparameters Do Not Match Table 2
   Table 2 specifies: encoder layers=4, hidden_dim=512, KAN_dim=128, dropout=0.1, batch_size=16, lr=1e-4, input_window=96. The code uses: layers=2, hidden_dim=64-256, KAN_dim=3-50, dropout=0.2-0.3, batch_size=32, lr=1e-3, window=10 or 192.

3. Normalization Mismatch
   The paper describes z-score normalization computed from the training set. The ETT, Traffic, and Weather dataloaders use min-max normalization (not z-score), and statistics are computed over the entire dataset (potential data leakage since there's no split).

4. Missing Value Handling
   The paper describes linear interpolation for missing values. The code uses dropna() (row deletion) instead. No interpolation code exists anywhere.

5. Single-Step vs Multi-Horizon Prediction
   The paper reports forecasting horizons of 96 and 192 steps. The code only predicts a single next time step (y = targets[idx + seq_length] — one scalar value).

6. Unused Configuration File
   configs/config.yaml exists with validation: enabled: false but is never loaded by any Python script (no import yaml exists). All hyperparameters are hardcoded.

7. Random Knowledge Vectors
   In the ablation experiment (kalformer.py line 33), the knowledge vector is torch.randn(...) — random noise rather than actual domain knowledge.

Could you please provide the actual code used to produce the results reported in Tables 3-5? The current repository appears to be an early development version rather than the final experimental code.

Thank you.

[dxpython]

Hello,

Thank you very much for your careful reading of our paper and for the detailed feedback on the repository. We sincerely appreciate your effort in examining the implementation and pointing out these inconsistencies.

You are correct that the current public repository does not fully reflect the final experimental pipeline used in the paper. The released code is an early research version intended for structural reference, and some experimental components (data split pipeline, evaluation scripts, and final hyperparameter settings) were not included during the initial cleanup and refactoring process before release.

We clarify the main points below:

• Data split and evaluation — The experiments in the paper used chronological 8:1:1 train/validation/test splits with a separate evaluation pipeline. These scripts were part of the internal experimental framework but were not included in the public release. We will update the repository to include them.

• Hyperparameters — Table 2 reports the final tuned configuration used in experiments. The released code contains default lightweight settings for demonstration and faster testing.

• Normalization — z-score normalization was applied in the final experiments using training-set statistics. The current dataloader implementation will be corrected to reflect this.

• Missing values — linear interpolation was used in the experimental pipeline; the current preprocessing module is incomplete and will be revised.

• Multi-horizon prediction — the public implementation currently exposes the single-step interface, while the full multi-horizon training code will be added.

• Configuration file — this is part of an unfinished refactoring stage and will be integrated.

• Knowledge vectors — the random vector is used only for preliminary ablation structure; the paper experiments use structured knowledge embeddings.

We acknowledge that the current repository may cause confusion regarding reproducibility, and we are preparing an updated version of the codebase that aligns with the experimental settings reported in the paper.

Thank you again for helping us improve the clarity and quality of the release.

Best regards,
For any questions, please email: anonymouslook5@outlook.com

[andermannfelix-max]

Thank you for the quick and transparent response, I really appreciate you taking the time to address the points raised.

Since I'm currently working on a benchmarking study that includes KALFormer, it would be very helpful to know roughly when the updated codebase might be available. Even a broad estimate (days / weeks) would help me plan accordingly.

Thanks in advance and best regards,
Felix Andermann

[dxpython]

Dear Felix,

Thank you for your message and for your interest in including KALFormer in your benchmarking study. I appreciate your careful attention to reproducibility and your continued engagement.

Due to ongoing research and project commitments, we are currently refining and consolidating the full experimental pipeline to ensure consistency with the methodology described in the paper. The updated codebase is expected to be released in approximately one month, although the exact timing may depend on internal validation and documentation completion.

If you have any questions regarding the model design, implementation details, or experimental settings in the meantime, please feel free to reach out at any time. I am happy to provide clarification where helpful.

Thank you again for your interest in our work.

[andermannfelix-max]

Thank you for the detailed response and for the timeline estimate. I fully understand that consolidating an experimental pipeline properly takes time, and I appreciate the effort you're putting into it.

I'm working on my own forecasting model at the moment, so the timing works perfectly. Looking forward to the updated release. I'll revisit this issue then.

Wishing you smooth progress on the refinements!