SBOLExplorer indexing can't complete

[cl117]

[cl117]

(jammy) sophia@sophia:~/git_repo/SBOLExplorer/flask$ ./start.sh
Starting SBOLExplorer
[2025-02-18T14:30:06.875676] SBOLExplorer started :)

• Serving Flask app 'explorer.py'
• Debug mode: off
  [2025-02-18T14:31:57.456205] Successfully updated config
  [2025-02-18T14:32:33.584183] Successfully updated config
  [2025-02-18T14:40:12.492237] ============ STARTING INDEXING ============

[2025-02-18T14:40:12.500192] ------------ Updating clusters ------------
[2025-02-18T14:40:12.500304] ******** Query for sequences ********
[2025-02-18T14:40:12.500415] Current endpoint: http://localhost:8890/sparql?
[2025-02-18T14:40:31.571553] ******** Query for sequences complete ********
[2025-02-18T14:40:31.806013] ******** Running uclust ********
[2025-02-18T14:40:31.813569]
[2025-02-18T14:40:31.813716] ******** Running uclust complete ********
[2025-02-18T14:40:31.880536] parts: 48548
[2025-02-18T14:40:31.880757] hits: 4142
[2025-02-18T14:40:31.880815] average hit identity: 99.96856591018833
[2025-02-18T14:40:31.880872] ------------ Successfully updated clusters ------------

[2025-02-18T14:40:32.050390] ------------ Updating pagerank ------------
[2025-02-18T14:40:32.050691] ******** Query for uris ********
[2025-02-18T14:40:32.050771] Current endpoint: http://localhost:8890/sparql?
[2025-02-18T14:40:36.364657] ******** Query for uris complete ********
[2025-02-18T14:40:36.680406] ******** Query for links ********
[2025-02-18T14:40:36.680623] Current endpoint: http://localhost:8890/sparql?
[2025-02-18T14:45:28.955955] ******** Query for links complete ********
[2025-02-18T14:45:30.373535] ******** Running pagerank ********
[2025-02-18T14:45:31.439385] Iteration 1: L1 norm delta is 0.8987276181233838
[2025-02-18T14:45:32.591029] Iteration 2: L1 norm delta is 0.33434115390245434
[2025-02-18T14:45:33.687040] Iteration 3: L1 norm delta is 0.12523916828775308
[2025-02-18T14:45:34.799756] Iteration 4: L1 norm delta is 0.05513771268106643
[2025-02-18T14:45:35.822345] Iteration 5: L1 norm delta is 0.028269005971071275
[2025-02-18T14:45:36.906171] Iteration 6: L1 norm delta is 0.017989438140787715
[2025-02-18T14:45:37.905077] Iteration 7: L1 norm delta is 0.012611744735979853
[2025-02-18T14:45:39.047883] Iteration 8: L1 norm delta is 0.00938623134254664
[2025-02-18T14:45:40.123957] Iteration 9: L1 norm delta is 0.0072761605760165715
[2025-02-18T14:45:41.211009] Iteration 10: L1 norm delta is 0.005783514395966579
[2025-02-18T14:45:42.186613] Iteration 11: L1 norm delta is 0.0046618348010533306
[2025-02-18T14:45:43.214547] Iteration 12: L1 norm delta is 0.00378682537541017
[2025-02-18T14:45:44.245506] Iteration 13: L1 norm delta is 0.0030886794528678543
[2025-02-18T14:45:45.324706] Iteration 14: L1 norm delta is 0.0025247301260282
[2025-02-18T14:45:46.391836] Iteration 15: L1 norm delta is 0.002066066784580232
[2025-02-18T14:45:47.499967] Iteration 16: L1 norm delta is 0.0016917125928484968
[2025-02-18T14:45:48.659961] Iteration 17: L1 norm delta is 0.0013855945741699455
[2025-02-18T14:45:49.844351] Iteration 18: L1 norm delta is 0.001135036672336263
[2025-02-18T14:45:50.876929] Iteration 19: L1 norm delta is 0.0009298533523861993
[2025-02-18T14:45:51.954692] Iteration 20: L1 norm delta is 0.0007617868405561738
[2025-02-18T14:45:52.997449] Iteration 21: L1 norm delta is 0.0006241062603277924
[2025-02-18T14:45:54.037489] Iteration 22: L1 norm delta is 0.0005113117737952053
[2025-02-18T14:45:55.052011] Iteration 23: L1 norm delta is 0.00041890296006135623
[2025-02-18T14:45:56.066245] Iteration 24: L1 norm delta is 0.0003431949051764873
[2025-02-18T14:45:57.212067] Iteration 25: L1 norm delta is 0.00028116927208739716
[2025-02-18T14:45:58.341035] Iteration 26: L1 norm delta is 0.00023035342922402954
[2025-02-18T14:45:59.364693] Iteration 27: L1 norm delta is 0.00018872157784439016
[2025-02-18T14:46:00.461432] Iteration 28: L1 norm delta is 0.00015461399674380642
[2025-02-18T14:46:01.476353] Iteration 29: L1 norm delta is 0.00012667086648109963
[2025-02-18T14:46:02.511659] Iteration 30: L1 norm delta is 0.0001037780914530889
[2025-02-18T14:46:03.522986] Iteration 31: L1 norm delta is 8.502290172581579e-05
[2025-02-18T14:46:03.523192] ******** Running pagerank complete ********
[2025-02-18T14:46:03.523251] ------------ Successfully updated pagerank ------------

[2025-02-18T14:46:04.334349] ------------ Updating index ------------
[2025-02-18T14:46:04.334531] ******** Query for parts ********
[2025-02-18T14:46:04.334608] Current endpoint: http://localhost:8890/sparql?
[2025-02-18T14:47:54.791669] [ERROR] exception when connecting: 500 Server Error: SPARQL Request Failed for url: http://localhost:8890/sparql?query=%0A++++PREFIX+rdf%3A+%3Chttp%3A%2F%2Fwww.w3.org%2F1999%2F02%2F22-rdf-syntax-ns%23%3E%0A++++PREFIX+dcterms%3A+%3Chttp%3A%2F%2Fpurl.org%2Fdc%2Fterms%2F%3E%0A++++PREFIX+dc%3A+%3Chttp%3A%2F%2Fpurl.org%2Fdc%2Felements%2F1.1%2F%3E%0A++++PREFIX+sbh%3A+%3Chttp%3A%2F%2Fwiki.synbiohub.org%2Fwiki%2FTerms%2Fsynbiohub%23%3E%0A++++PREFIX+synbiohub%3A+%3Chttp%3A%2F%2Fsynbiohub.org%23%3E%0A++++PREFIX+igem%3A+%3Chttp%3A%2F%2Fwiki.synbiohub.org%2Fwiki%2FTerms%2Figem%23%3E%0A++++PREFIX+prov%3A+%3Chttp%3A%2F%2Fwww.w3.org%2Fns%2Fprov%23%3E%0A++++PREFIX+sbol2%3A+%3Chttp%3A%2F%2Fsbols.org%2Fv2%23%3E%0A++++PREFIX+xsd%3A+%3Chttp%3A%2F%2Fwww.w3.org%2F2001%2FXMLSchema%23%3E%0A++++PREFIX+rdfs%3A+%3Chttp%3A%2F%2Fwww.w3.org%2F2000%2F01%2Frdf-schema%23%3E%0A++++PREFIX+purl%3A+%3Chttp%3A%2F%2Fpurl.obolibrary.org%2Fobo%2F%3E%0A++++PREFIX+ncbi%3A+%3Chttp%3A%2F%2Fwww.ncbi.nlm.nih.gov%23%3E%0A+++++%0A++++SELECT+DISTINCT%0A++++++++%3Fsubject%0A++++++++%3FdisplayId%0A++++++++%3Fversion%0A++++++++%3Fname%0A++++++++%3Fdescription%0A++++++++%3Ftype%0A++++++++%3Fgraph%0A++++++++%3Frole%0A++++++++%3Fsboltype%0A++++%0A++++WHERE+%7B%0A++++%0A++++++++%3Fsubject+a+%3Ftype+.%0A++++++++%3Fsubject+sbh%3AtopLevel+%3Fsubject+.%0A++++++++GRAPH+%3Fgraph+%7B+%3Fsubject+%3Fa+%3Ft+%7D+.%0A++++++++OPTIONAL+%7B+%3Fsubject+sbol2%3AdisplayId+%3FdisplayId+.+%7D%0A++++++++OPTIONAL+%7B+%3Fsubject+sbol2%3Aversion+%3Fversion+.+%7D%0A++++++++OPTIONAL+%7B+%3Fsubject+dcterms%3Atitle+%3Fname+.+%7D%0A++++++++OPTIONAL+%7B+%3Fsubject+dcterms%3Adescription+%3Fdescription+.+%7D%0A++++++++OPTIONAL+%7B+%3Fsubject+sbol2%3Arole+%3Frole+.+%7D%0A++++++++OPTIONAL+%7B+%3Fsubject+sbol2%3Atype+%3Fsboltype+.+%7D%0A++++%7D+%0A+++++OFFSET+110000+LIMIT+10000&default-graph-uri=
[2025-02-18T14:47:54.791971] [ERROR] failed querying: http://localhost:8890/sparql? - Local SynBioHub isn't responding
[2025-02-18T14:47:54.986193] ******** Query for parts complete ********
[2025-02-18T14:47:54.986543] ******** Adding parts to new index ********
[2025-02-18T14:47:55.070732] Index already exists -> deleting
[2025-02-18T14:47:55.573505] Index created
[2025-02-18T14:47:55.573844] Bulk indexing
[2025-02-18T14:47:55.574019] Bulk indexing complete
[2025-02-18T14:47:55.574194] ******** Finished adding 0 parts to index ********
[2025-02-18T14:47:55.574355] ------------ Successfully updated index ------------

[2025-02-18T14:47:55.575473] Cache cleared
[2025-02-18T14:47:55.576207] ============ INDEXING COMPLETED ============

[cl117]

The issue was resolved by modifying parameters in virtuoso.ini. In the experiment, after reducing the limit to 1000, the offset continued to increment until it exceeded 112,000, at which point the query failed. This indicates that the bottleneck is not the limit itself but rather Virtuoso's maximum execution time constraint.

[cl117]

Question: can't know which virtuoso parameters I changed, can't see the results from Docker environment. Test by writing the 3 environment in the docker-compose file and rerun the the experimemt, but don't remove the volume.

[cl117]

Virtuoso .ini file:
MaxQueryExecutionTime = 3600 # Allow longer execution (default is 60)
MaxQueryCostEstimationTime = 1200 # Increase estimation time
ResultSetMaxRows = 1000000 # Allow returning more rows

[cjmyers]

Still stuck with bulk indexing. Shutting explorer back off