Skip to content

Commit 96c203b

Browse files
ErinWeisbartErinWeisbart
committed
deploy: 49c1af8 
1 parent ad988b2 commit 96c203b

File tree

3 files changed

+42
-33
lines changed

3 files changed

+42
-33
lines changed

‎_sources/config_examples.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ Our internal configurations for each pipeline are as follows:
3636
| SSH_KEY_NAME | 'YOURPEM.pem' | 'YOURPEM.pem' | 'YOURPEM.pem' | 'YOURPEM.pem' | 'YOURPEM.pem' | |
3737
| AWS_BUCKET | 'BUCKET' | 'BUCKET' | 'BUCKET' | 'BUCKET' | 'BUCKET' | Usually a bucket in the account that is running DCP. |
3838
| SOURCE_BUCKET | 'BUCKET' | 'BUCKET' | 'BUCKET' | 'BUCKET' | 'BUCKET' | Can be a public bucket like cellpainting-gallery. |
39+
| WORKSPACE_BUCKET | 'BUCKET' | 'BUCKET' | 'BUCKET' | 'BUCKET' | 'BUCKET' | If reading images from a public bucket, you might still want to read metadata from your bucket. |
3940
| DESTINATION_BUCKET | 'BUCKET' | 'BUCKET' | 'BUCKET' | 'BUCKET' | 'BUCKET' | Usually a bucket in the account that is running DCP. |
4041
| UPLOAD_FLAGS | '' | '' | '' | '' | '' | |
4142
| ECS_CLUSTER | 'default' | 'default' | 'default' | 'default' | 'default' | Most of the time we all just use the default cluster but if there are multiple jobs being run at once you can create your own cluster by changing default to YOURNAME so that the correct dockers go on the correct machines. |

‎config_examples.html‎

Lines changed: 40 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -514,255 +514,263 @@ <h1>config.py configuration examples<a class="headerlink" href="#config-py-confi
514514
<td><p>‘BUCKET’</p></td>
515515
<td><p>Can be a public bucket like cellpainting-gallery.</p></td>
516516
</tr>
517-
<tr class="row-even"><td><p>DESTINATION_BUCKET</p></td>
517+
<tr class="row-even"><td><p>WORKSPACE_BUCKET</p></td>
518+
<td><p>‘BUCKET’</p></td>
519+
<td><p>‘BUCKET’</p></td>
520+
<td><p>‘BUCKET’</p></td>
521+
<td><p>‘BUCKET’</p></td>
522+
<td><p>‘BUCKET’</p></td>
523+
<td><p>If reading images from a public bucket, you might still want to read metadata from your bucket.</p></td>
524+
</tr>
525+
<tr class="row-odd"><td><p>DESTINATION_BUCKET</p></td>
518526
<td><p>‘BUCKET’</p></td>
519527
<td><p>‘BUCKET’</p></td>
520528
<td><p>‘BUCKET’</p></td>
521529
<td><p>‘BUCKET’</p></td>
522530
<td><p>‘BUCKET’</p></td>
523531
<td><p>Usually a bucket in the account that is running DCP.</p></td>
524532
</tr>
525-
<tr class="row-odd"><td><p>UPLOAD_FLAGS</p></td>
533+
<tr class="row-even"><td><p>UPLOAD_FLAGS</p></td>
526534
<td><p>‘’</p></td>
527535
<td><p>‘’</p></td>
528536
<td><p>‘’</p></td>
529537
<td><p>‘’</p></td>
530538
<td><p>‘’</p></td>
531539
<td><p></p></td>
532540
</tr>
533-
<tr class="row-even"><td><p>ECS_CLUSTER</p></td>
541+
<tr class="row-odd"><td><p>ECS_CLUSTER</p></td>
534542
<td><p>‘default’</p></td>
535543
<td><p>‘default’</p></td>
536544
<td><p>‘default’</p></td>
537545
<td><p>‘default’</p></td>
538546
<td><p>‘default’</p></td>
539547
<td><p>Most of the time we all just use the default cluster but if there are multiple jobs being run at once you can create your own cluster by changing default to YOURNAME so that the correct dockers go on the correct machines.</p></td>
540548
</tr>
541-
<tr class="row-odd"><td><p>CLUSTER_MACHINES</p></td>
549+
<tr class="row-even"><td><p>CLUSTER_MACHINES</p></td>
542550
<td><p>100-200</p></td>
543551
<td><p>number of plates / CPUs and rounded up</p></td>
544552
<td><p>25-100</p></td>
545553
<td><p>25-100</p></td>
546554
<td><p>100-200</p></td>
547555
<td><p>AWS has limits on the number of machines you can request at a time. 200 is generally the largest we request for a single job to ensure there is some capacity for other users in the team.</p></td>
548556
</tr>
549-
<tr class="row-even"><td><p>TASKS_PER_MACHINE</p></td>
557+
<tr class="row-odd"><td><p>TASKS_PER_MACHINE</p></td>
550558
<td><p>1</p></td>
551559
<td><p>1</p></td>
552560
<td><p>1</p></td>
553561
<td><p>1</p></td>
554562
<td><p>1</p></td>
555563
<td><p></p></td>
556564
</tr>
557-
<tr class="row-odd"><td><p>MACHINE_TYPE</p></td>
565+
<tr class="row-even"><td><p>MACHINE_TYPE</p></td>
558566
<td><p>[‘c5.xlarge’]</p></td>
559567
<td><p>[‘c5.xlarge’]</p></td>
560568
<td><p>[‘c5.xlarge’]</p></td>
561569
<td><p>[‘c5.xlarge’]</p></td>
562570
<td><p>[‘c5.xlarge’]</p></td>
563571
<td><p>Historically we have used m4.xlarge and then m5.xlarge however very recently we have been having a hard time getting m class machines so we have switched to c class. Note that they have different memory sizes so you need to make sure MEMORY is set correctly if changing between classes.</p></td>
564572
</tr>
565-
<tr class="row-even"><td><p>MACHINE_PRICE</p></td>
573+
<tr class="row-odd"><td><p>MACHINE_PRICE</p></td>
566574
<td><p>.20</p></td>
567575
<td><p>.20</p></td>
568576
<td><p>.20</p></td>
569577
<td><p>.20</p></td>
570578
<td><p>.20</p></td>
571579
<td><p>Will be different for different size/classes of machines.</p></td>
572580
</tr>
573-
<tr class="row-odd"><td><p>EBS_VOL_SIZE (if using S3 mounted as a file system)</p></td>
581+
<tr class="row-even"><td><p>EBS_VOL_SIZE (if using S3 mounted as a file system)</p></td>
574582
<td><p>22</p></td>
575583
<td><p>22</p></td>
576584
<td><p>22</p></td>
577585
<td><p>22</p></td>
578586
<td><p>22</p></td>
579587
<td><p>Files are read directly off of S3, mounted as a file system when <code class="docutils literal notranslate"><span class="pre">DOWNLOAD_FILES</span> <span class="pre">=</span> <span class="pre">False</span></code>.</p></td>
580588
</tr>
581-
<tr class="row-even"><td><p>EBS_VOL_SIZE (if downloading files)</p></td>
589+
<tr class="row-odd"><td><p>EBS_VOL_SIZE (if downloading files)</p></td>
582590
<td><p>22</p></td>
583591
<td><p>200</p></td>
584592
<td><p>22</p></td>
585593
<td><p>22</p></td>
586594
<td><p>40</p></td>
587595
<td><p>Files are downloaded to the EBS volume when <code class="docutils literal notranslate"><span class="pre">DOWNLOAD_FILES</span> <span class="pre">=</span> <span class="pre">True</span></code>.</p></td>
588596
</tr>
589-
<tr class="row-odd"><td><p>DOWNLOAD_FILES</p></td>
597+
<tr class="row-even"><td><p>DOWNLOAD_FILES</p></td>
590598
<td><p>‘False’</p></td>
591599
<td><p>‘False’</p></td>
592600
<td><p>‘False’</p></td>
593601
<td><p>‘False’</p></td>
594602
<td><p>‘False’</p></td>
595603
<td><p></p></td>
596604
</tr>
597-
<tr class="row-even"><td><p>ASSIGN_IP</p></td>
605+
<tr class="row-odd"><td><p>ASSIGN_IP</p></td>
598606
<td><p>‘False’</p></td>
599607
<td><p>‘False’</p></td>
600608
<td><p>‘False’</p></td>
601609
<td><p>‘False’</p></td>
602610
<td><p>‘False’</p></td>
603611
<td><p></p></td>
604612
</tr>
605-
<tr class="row-odd"><td><p>DOCKER_CORES</p></td>
613+
<tr class="row-even"><td><p>DOCKER_CORES</p></td>
606614
<td><p>4</p></td>
607615
<td><p>4</p></td>
608616
<td><p>4</p></td>
609617
<td><p>4</p></td>
610618
<td><p>3</p></td>
611619
<td><p>If using c class machines and large images (2k + pixels) then you might need to reduce this number.</p></td>
612620
</tr>
613-
<tr class="row-even"><td><p>CPU_SHARES</p></td>
621+
<tr class="row-odd"><td><p>CPU_SHARES</p></td>
614622
<td><p>DOCKER_CORES * 1024</p></td>
615623
<td><p>DOCKER_CORES * 1024</p></td>
616624
<td><p>DOCKER_CORES * 1024</p></td>
617625
<td><p>DOCKER_CORES * 1024</p></td>
618626
<td><p>DOCKER_CORES * 1024</p></td>
619627
<td><p>We never change this.</p></td>
620628
</tr>
621-
<tr class="row-odd"><td><p>MEMORY</p></td>
629+
<tr class="row-even"><td><p>MEMORY</p></td>
622630
<td><p>7500</p></td>
623631
<td><p>7500</p></td>
624632
<td><p>7500</p></td>
625633
<td><p>7500</p></td>
626634
<td><p>7500</p></td>
627635
<td><p>This must match your machine type. m class use 15000, c class use 7500.</p></td>
628636
</tr>
629-
<tr class="row-even"><td><p>SECONDS_TO_START</p></td>
637+
<tr class="row-odd"><td><p>SECONDS_TO_START</p></td>
630638
<td><p>60</p></td>
631639
<td><p>3*60</p></td>
632640
<td><p>60</p></td>
633641
<td><p>3*60</p></td>
634642
<td><p>3*60</p></td>
635643
<td><p></p></td>
636644
</tr>
637-
<tr class="row-odd"><td><p>SQS_QUEUE_NAME</p></td>
645+
<tr class="row-even"><td><p>SQS_QUEUE_NAME</p></td>
638646
<td><p>APP_NAME + ‘Queue’</p></td>
639647
<td><p>APP_NAME + ‘Queue’</p></td>
640648
<td><p>APP_NAME + ‘Queue’</p></td>
641649
<td><p>APP_NAME + ‘Queue’</p></td>
642650
<td><p>APP_NAME + ‘Queue’</p></td>
643651
<td><p>We never change this.</p></td>
644652
</tr>
645-
<tr class="row-even"><td><p>SQS_MESSAGE_VISIBILITY</p></td>
653+
<tr class="row-odd"><td><p>SQS_MESSAGE_VISIBILITY</p></td>
646654
<td><p>3*60</p></td>
647655
<td><p>240*60</p></td>
648656
<td><p>15*60</p></td>
649657
<td><p>10*60</p></td>
650658
<td><p>120*60</p></td>
651659
<td><p>About how long you expect a job to take * 1.5 in seconds</p></td>
652660
</tr>
653-
<tr class="row-odd"><td><p>SQS_DEAD_LETTER_QUEUE</p></td>
661+
<tr class="row-even"><td><p>SQS_DEAD_LETTER_QUEUE</p></td>
654662
<td><p>‘YOURNAME_DEADMESSAGES’</p></td>
655663
<td><p>‘YOURNAME_DEADMESSAGES’</p></td>
656664
<td><p>‘YOURNAME_DEADMESSAGES’</p></td>
657665
<td><p>‘YOURNAME_DEADMESSAGES’</p></td>
658666
<td><p>‘YOURNAME_DEADMESSAGES’</p></td>
659667
<td><p></p></td>
660668
</tr>
661-
<tr class="row-even"><td><p>JOB_RETRIES</p></td>
669+
<tr class="row-odd"><td><p>JOB_RETRIES</p></td>
662670
<td><p>3</p></td>
663671
<td><p>3</p></td>
664672
<td><p>3</p></td>
665673
<td><p>3</p></td>
666674
<td><p>3</p></td>
667675
<td><p></p></td>
668676
</tr>
669-
<tr class="row-odd"><td><p>AUTO_MONITOR</p></td>
677+
<tr class="row-even"><td><p>AUTO_MONITOR</p></td>
670678
<td><p>‘True’</p></td>
671679
<td><p>‘True’</p></td>
672680
<td><p>‘True’</p></td>
673681
<td><p>‘True’</p></td>
674682
<td><p>‘True’</p></td>
675683
<td><p>Can be turned off if manually running Monitor.</p></td>
676684
</tr>
677-
<tr class="row-even"><td><p>CREATE_DASHBOARD</p></td>
685+
<tr class="row-odd"><td><p>CREATE_DASHBOARD</p></td>
678686
<td><p>‘True’</p></td>
679687
<td><p>‘True’</p></td>
680688
<td><p>‘True’</p></td>
681689
<td><p>‘True’</p></td>
682690
<td><p>‘True’</p></td>
683691
<td><p></p></td>
684692
</tr>
685-
<tr class="row-odd"><td><p>CLEAN_DASHBOARD</p></td>
693+
<tr class="row-even"><td><p>CLEAN_DASHBOARD</p></td>
686694
<td><p>‘True’</p></td>
687695
<td><p>‘True’</p></td>
688696
<td><p>‘True’</p></td>
689697
<td><p>‘True’</p></td>
690698
<td><p>‘True’</p></td>
691699
<td><p></p></td>
692700
</tr>
693-
<tr class="row-even"><td><p>CHECK_IF_DONE_BOOL</p></td>
701+
<tr class="row-odd"><td><p>CHECK_IF_DONE_BOOL</p></td>
694702
<td><p>‘True’</p></td>
695703
<td><p>‘True’</p></td>
696704
<td><p>‘True’</p></td>
697705
<td><p>‘True’</p></td>
698706
<td><p>‘True’</p></td>
699707
<td><p>Can be turned off if wanting to overwrite old data.</p></td>
700708
</tr>
701-
<tr class="row-odd"><td><p>EXPECTED_NUMBER_FILES</p></td>
709+
<tr class="row-even"><td><p>EXPECTED_NUMBER_FILES</p></td>
702710
<td><p>1 (an image)</p></td>
703711
<td><p>number channels + 1 (an .npy for each channel and isdone)</p></td>
704712
<td><p>3 (Experiment.csv, Image.csv, and isdone)</p></td>
705713
<td><p>1 (an image)</p></td>
706714
<td><p>5 (Experiment, Image, Cells, Nuclei, and Cytoplasm .csvs)</p></td>
707715
<td><p>Better to underestimate than overestimate.</p></td>
708716
</tr>
709-
<tr class="row-even"><td><p>MIN_FILE_SIZE_BYTES</p></td>
717+
<tr class="row-odd"><td><p>MIN_FILE_SIZE_BYTES</p></td>
710718
<td><p>1</p></td>
711719
<td><p>1</p></td>
712720
<td><p>1</p></td>
713721
<td><p>1</p></td>
714722
<td><p>1</p></td>
715723
<td><p>Count files of any size.</p></td>
716724
</tr>
717-
<tr class="row-odd"><td><p>NECESSARY_STRING</p></td>
725+
<tr class="row-even"><td><p>NECESSARY_STRING</p></td>
718726
<td><p>‘’</p></td>
719727
<td><p>‘’</p></td>
720728
<td><p>‘’</p></td>
721729
<td><p>‘’</p></td>
722730
<td><p>‘’</p></td>
723731
<td><p>Not necessary for standard workflows.</p></td>
724732
</tr>
725-
<tr class="row-even"><td><p>ALWAYS_CONTINUE</p></td>
733+
<tr class="row-odd"><td><p>ALWAYS_CONTINUE</p></td>
726734
<td><p>‘False’</p></td>
727735
<td><p>‘False’</p></td>
728736
<td><p>‘False’</p></td>
729737
<td><p>‘False’</p></td>
730738
<td><p>‘False’</p></td>
731739
<td><p>Use with caution.</p></td>
732740
</tr>
733-
<tr class="row-odd"><td><p>USE_PLUGINS</p></td>
741+
<tr class="row-even"><td><p>USE_PLUGINS</p></td>
734742
<td><p>‘False’</p></td>
735743
<td><p>‘False’</p></td>
736744
<td><p>‘False’</p></td>
737745
<td><p>‘False’</p></td>
738746
<td><p>‘False’</p></td>
739747
<td><p>Not necessary for standard workflows.</p></td>
740748
</tr>
741-
<tr class="row-even"><td><p>UPDATE_PLUGINS</p></td>
749+
<tr class="row-odd"><td><p>UPDATE_PLUGINS</p></td>
742750
<td><p>‘False’</p></td>
743751
<td><p>‘False’</p></td>
744752
<td><p>‘False’</p></td>
745753
<td><p>‘False’</p></td>
746754
<td><p>‘False’</p></td>
747755
<td><p>Not necessary for standard workflows.</p></td>
748756
</tr>
749-
<tr class="row-odd"><td><p>PLUGINS_COMMIT</p></td>
757+
<tr class="row-even"><td><p>PLUGINS_COMMIT</p></td>
750758
<td><p>‘’</p></td>
751759
<td><p>‘’</p></td>
752760
<td><p>‘’</p></td>
753761
<td><p>‘’</p></td>
754762
<td><p>‘’</p></td>
755763
<td><p>Not necessary for standard workflows.</p></td>
756764
</tr>
757-
<tr class="row-even"><td><p>INSTALL_REQUIREMENTS</p></td>
765+
<tr class="row-odd"><td><p>INSTALL_REQUIREMENTS</p></td>
758766
<td><p>‘False’</p></td>
759767
<td><p>‘False’</p></td>
760768
<td><p>‘False’</p></td>
761769
<td><p>‘False’</p></td>
762770
<td><p>‘False’</p></td>
763771
<td><p>Not necessary for standard workflows.</p></td>
764772
</tr>
765-
<tr class="row-odd"><td><p>REQUIREMENTS_FILE</p></td>
773+
<tr class="row-even"><td><p>REQUIREMENTS_FILE</p></td>
766774
<td><p>‘’</p></td>
767775
<td><p>‘’</p></td>
768776
<td><p>‘’</p></td>

0 commit comments

Comments
 (0)