Micom couldn't pick up column names when building community model

[suetli19]

Hi there,

I tried to input my data frame to build community model but encounter error which shows:

IndexError                                Traceback (most recent call last)
Cell In[89], line 1
----> 1 manifest = build(
      2     taxonomy,
      3     model_db="./agora201_gtdb207_species_1.qza",
      4     out_folder="br103_model",
      5     threads=2
      6 )

File ~/LUSTRE/miniconda3/lib/python3.9/site-packages/micom/workflows/build.py:111, in build(taxonomy, model_db, out_folder, cutoff, threads, solver)
    106 out_path = pd.Series({s: os.path.join(out_folder, s + ".pickle") for s in samples})
    107 args = [
    108     [s, taxonomy[taxonomy.sample_id == s], model_db, out_path[s], cutoff, solver]
    109     for s in samples
    110 ]
--> 111 res = workflow(build_and_save, args, threads)
    112 metrics = pd.concat(res)
    113 taxonomy = (
    114     taxonomy.groupby("sample_id")
    115     .apply(_reduce_group)
    116     .dropna(axis=1)
    117     .reset_index(drop=True)
    118 )

File ~/LUSTRE/miniconda3/lib/python3.9/site-packages/micom/workflows/core.py:55, in workflow(func, args, threads, description, progress)
     53         if progress:
     54             it = track(it, total=len(args), description="Running")
---> 55         results = list(it)
     56 finally:
     57     pool.close()

File ~/LUSTRE/miniconda3/lib/python3.9/site-packages/rich/progress.py:173, in track(sequence, description, total, completed, auto_refresh, console, transient, get_time, refresh_per_second, style, complete_style, finished_style, pulse_style, update_period, disable, show_speed)
    162 progress = Progress(
    163     *columns,
    164     auto_refresh=auto_refresh,
   (...)
    169     disable=disable,
    170 )
    172 with progress:
--> 173     yield from progress.track(
    174         sequence,
    175         total=total,
    176         completed=completed,
    177         description=description,
    178         update_period=update_period,
    179     )

File ~/LUSTRE/miniconda3/lib/python3.9/site-packages/rich/progress.py:1225, in Progress.track(self, sequence, total, completed, task_id, description, update_period)
   1223 if self.live.auto_refresh:
   1224     with _TrackThread(self, task_id, update_period) as track_thread:
-> 1225         for value in sequence:
   1226             yield value
   1227             track_thread.completed += 1

File ~/LUSTRE/miniconda3/lib/python3.9/multiprocessing/pool.py:870, in IMapIterator.next(self, timeout)
    868 if success:
    869     return value
--> 870 raise value

IndexError: index 0 is out of bounds for axis 0 with size 0



**This is my taxonomy columns when I check for "taxonomy.columns":**

Index(['sample_id', 'id', 'abundance', 'species', 'genus', 'family', 'order',
       'class', 'phylum'],
      dtype='object')

Could you help me to figure what has went wrong? Thank you!

[cdiener]

I don't see really see where it complains about missing columns. Hard to say without seeing the actual data but one problem is that this table will not match to any models in the database. By default, it will always use all taxonomic ranks in the table, so phylum to species in your case. The NA entries will not match the database. The species names are also expected to include the Genus for GTDB data because this is how the GTDB does it. So it should be s__Clostridium butyricum not s__butyricum.

[suetli19]

I don't see really see where it complains about missing columns. Hard to say without seeing the actual data but one problem is that this table will not match to any models in the database. By default, it will always use all taxonomic ranks in the table, so phylum to species in your case. The NA entries will not match the database. The species names are also expected to include the Genus for GTDB data because this is how the GTDB does it. So it should be s__Clostridium butyricum not s__butyricum.

Thanks for your reply!

I have changed the species names to include genus.

However, I still encounter the same error message that says, "IndexError: index 0 is out of bounds for axis 0 with size 0"

I am wondering if there is an issue with the database that I use to call the function, if you could help me to have a look at the code:

manifest = build(
df,
model_db="./agora201_gtdb207_species_1.qza",
out_folder="br103_model",
threads=2)

I have attached the manifest file from the database here for your reference.

Thank you for your help.

manifest.csv

[cdiener]

I think it is still not matching because of the phylum names. Our AGORA2 database still uses GTDB207 which has the old phyla, so for instance p__Actinobacteriota for B. longum. Usually your classifications and the DB should use the same version if possible. In your case the easiest workaround is to drop the additional ranks and only leave the species in your table. It should work with that. I will try to prep a new DB release for MICOM soon.

In case this is not working you could also run the build() step with threads=1 which will give are more informative error message.

[suetli19]

I think it is still not matching because of the phylum names. Our AGORA2 database still uses GTDB207 which has the old phyla, so for instance p__Actinobacteriota for B. longum. Usually your classifications and the DB should use the same version if possible. In your case the easiest workaround is to drop the additional ranks and only leave the species in your table. It should work with that. I will try to prep a new DB release for MICOM soon.

In case this is not working you could also run the build() step with threads=1 which will give are more informative error message.

I removed the other taxa ranks and only retained the species and re-run the build function with threads=1, but it still did not work. Here's the error message:

IndexError Traceback (most recent call last)
/tmp/ipykernel_1319682/4016778043.py in ?()
----> 5 from micom.workflows import build
6 from micom import Community
7 import pandas as pd
8

~/LUSTRE/miniconda3/lib/python3.9/site-packages/micom/workflows/build.py in ?(taxonomy, model_db, out_folder, cutoff, threads, solver)
107 args = [
108 [s, taxonomy[taxonomy.sample_id == s], model_db, out_path[s], cutoff, solver]
109 for s in samples
110 ]
--> 111 res = workflow(build_and_save, args, threads)
112 metrics = pd.concat(res)
113 taxonomy = (
114 taxonomy.groupby("sample_id")

~/LUSTRE/miniconda3/lib/python3.9/site-packages/micom/workflows/core.py in ?(func, args, threads, description, progress)
40 if threads == 1:
41 it = map(func, args)
42 if progress:
43 it = track(it, total=len(args), description=description)
---> 44 return list(it)
45
46 # We don't use the context manager because of
47 # https://pytest-cov.readthedocs.io/en/latest/subprocess-support.html

~/LUSTRE/miniconda3/lib/python3.9/site-packages/rich/progress.py in ?(sequence, description, total, completed, auto_refresh, console, transient, get_time, refresh_per_second, style, complete_style, finished_style, pulse_style, update_period, disable, show_speed)
169 disable=disable,
170 )
171
172 with progress:
--> 173 yield from progress.track(
174 sequence,
175 total=total,
176 completed=completed,

~/LUSTRE/miniconda3/lib/python3.9/site-packages/rich/progress.py in ?(self, sequence, total, completed, task_id, description, update_period)
1223 if self.live.auto_refresh:
1224 with _TrackThread(self, task_id, update_period) as track_thread:
1225 for value in sequence:
1226 yield value
-> 1227 track_thread.completed += 1
1228 else:
1229 advance = self.advance
1230 refresh = self.refresh

~/LUSTRE/miniconda3/lib/python3.9/site-packages/micom/workflows/build.py in ?(args)
28 s, tax, db, out, cutoff, solver = args
29 if os.path.exists(out):
30 com = load_pickle(out)
31 else:
---> 32 com = Community(
33 tax, model_db=db, id=s, progress=False, rel_threshold=cutoff, solver=solver
34 )
35 com.to_pickle(out)

~/LUSTRE/miniconda3/lib/python3.9/site-packages/micom/community.py in ?(self, taxonomy, model_db, id, name, rel_threshold, solver, progress, max_exchange, mass)
209 for r in RANKS[0 : (RANKS.index(rank) + 1)]
210 if r in taxonomy.columns and r in manifest.columns
211 ]
212 manifest = manifest[keep_cols + ["file"]]
--> 213 taxonomy = unify_rank_prefixes(taxonomy, manifest)
214 merged = pd.merge(taxonomy, manifest, on=keep_cols)
215
216 self.__db_metrics = pd.Series(

~/LUSTRE/miniconda3/lib/python3.9/site-packages/micom/taxonomy.py in ?(taxonomy, manifest)
123 -------
124 tuple of pandas.DataFrame
125 The taxonomy with adjusted taxa names consistent with the database.
126 """
--> 127 tax_prefixes = rank_prefixes(taxonomy)
128 db_prefixes = rank_prefixes(manifest)
129 ranks = tax_prefixes.index[tax_prefixes.index.isin(db_prefixes.index)]
130 if all(tax_prefixes[ranks] == db_prefixes[ranks]):

~/LUSTRE/miniconda3/lib/python3.9/site-packages/micom/taxonomy.py in ?(manifest)
102 The detected prefix for each taxonomic rank in the manifest.
103 """
104 ranks = [c for c in manifest.columns if c.lower() in RANKS]
105 prefixes = pd.Series(
--> 106 {r: manifest[r].str.extract(r"^([a-z]__)").iloc[0, 0] for r in ranks}
107 )
108
109 return prefixes

~/LUSTRE/miniconda3/lib/python3.9/site-packages/micom/taxonomy.py in ?(.0)
--> 106 def rank_prefixes(manifest: pd.DataFrame) -> pd.Series:
107 """Get the used prefixes for taxonomic ranks.
108
109 Arguments

~/LUSTRE/miniconda3/lib/python3.9/site-packages/pandas/core/indexing.py in ?(self, key)
1179 if type(key) is tuple:
1180 key = tuple(list(x) if is_iterator(x) else x for x in key)
1181 key = tuple(com.apply_if_callable(x, self.obj) for x in key)
1182 if self._is_scalar_access(key):
-> 1183 return self.obj._get_value(*key, takeable=self._takeable)
1184 return self._getitem_tuple(key)
1185 else:
1186 # we by definition only have the 0th axis

~/LUSTRE/miniconda3/lib/python3.9/site-packages/pandas/core/frame.py in ?(self, index, col, takeable)
4208 self.columns._index_as_unique; Caller is responsible for checking.
4209 """
4210 if takeable:
4211 series = self._ixs(col, axis=1)
-> 4212 return series._values[index]
4213
4214 series = self._get_item_cache(col)
4215 engine = self.index._engine

IndexError: index 0 is out of bounds for axis 0 with size 0

I found out the database used to classify my taxa is GTDB220 but I saw that the most recent database in the micom website is up until GTDB207. Here's the database link that I referred to (https://zenodo.org/records/7739096). I'm wondering if there are newer DB versions for MICOM? Is there any other alternative solution for this?

Thanks!

[cdiener]

Hmm, okay I would not have expected the error there. Hard to debug without seeing the taxonomy table. Could you maybe share the last version with me (confidentially) at mail [at] dienerlab.com? Thanks.

[suetli19]

mail [at] dienerlab.com

Thanks for offering to help me debug. I have now sent an email to you and I'm wondering if you received it?

[cdiener]

No, unfortunately I have not received anything yet.

EDIT: Got it now, thanks!

[cdiener]

Hi the issue was that two of the samples summed to a zero total abundance: bb0216, bb0345. For now you can fix it by excluding zero abundances from the table (taxonomy = taxonomy[taxonomy.abundance > 0]).

The next MICOM release will check for those automatically and also report some more on the matching failures:

[11/18/25 10:13:32] WARNING  The following samples sum to a zero abundance and will be excluded: bb0216, bb0345                        build.py:107
Running ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:16
[11/18/25 10:13:49] WARNING  The following samples had no taxon matches in the model database and will be excluded. We recommend to    build.py:133
                             verify that the taxon names and  ranks match the database and version. Missing samples: bb0028, bb0076,               
                             bb0106, bb0196, bb0198, bb0312, bb0428, bb0475, bb0659 . 

[suetli19]

Hi the issue was that two of the samples summed to a zero total abundance: bb0216, bb0345. For now you can fix it by excluding zero abundances from the table (taxonomy = taxonomy[taxonomy.abundance > 0]).

The next MICOM release will check for those automatically and also report some more on the matching failures:

[11/18/25 10:13:32] WARNING  The following samples sum to a zero abundance and will be excluded: bb0216, bb0345                        build.py:107
Running ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:16
[11/18/25 10:13:49] WARNING  The following samples had no taxon matches in the model database and will be excluded. We recommend to    build.py:133
                             verify that the taxon names and  ranks match the database and version. Missing samples: bb0028, bb0076,               
                             bb0106, bb0196, bb0198, bb0312, bb0428, bb0475, bb0659 . 

Hi,

Thanks, it works fine now!