migrate VMs from ORC

[mbjones]

VMs that need to be migrated from ORC to UCSB:

• cn-orc-1 (production, 3.6T disk/1.3TB in use, 70G memory, 10 CPUs): Anacapa
• search-orc-1 (production, ~1TB disk/6GB in use, 4GB memory, 2 CPUs): Anacapa
• mn-orc-1 (production, 2.2T disk/1.6TB in use, 8G memory, 3 CPUs): Anacapa

Good to migrate if possible, but not as critical:

• cn-sandbox-orc-1(test, 1.2T disk/1.1T in use, 39G memory, 5 CPUs): NHDC
• cn-stage-orc-1 (test, 3.7T disk/1.2T in use, 39G memory, 5 CPUs): NHDC

@nickatnceas let's discuss placement of these. If possible, I'd like to move some of these to Anacapa, and others to NHDC (as indicated above). @taojing2002 and @datadavev can coordinate the moves on the DataONE side.

[nickatnceas]

@mbjones as of right now we don't have any VM hosts at Anacapa with the required specs, but we can move an older host or two from the NHDC.

The two VM hosts we are running at Anacapa (Pluto and Io) are 4 core R330/R340 1U servers with limited upgrade potential.

One note I have is that since Anacapa will be routed through campus, most of the planned campus network outages (like the few we had recently related to campus WiFi) will take down both the NHDC and Anacapa at the same time.

[taojing2002]

We have some configurations (particularly on CNs) using the domain names. It will be great we can keep those domain names.

[mbjones]

@taojing2002 Yes, the domain names can stay the same.

We plan to migrate the 3 production VMs to Poseidon at NHDC, and then move that host to Anacapa once the transfer is complete. @nickatnceas will coordinate with @taojing2002 and @datadavev on shutting down services at needed times.

The other two non-production VMs will go to NHDC.

[nickatnceas]

I will need to reconfigure the hardware raid config on Poseidon, which will involve reinstalling the OS. This might take a day or two before we can start transferring data.

[nickatnceas]

I have server progress tracking at https://github.nceas.ucsb.edu/NCEAS/Computing/issues/106

I should be able to start initial (online) rsyncs tomorrow, and then offline migrations Thursday or Friday.

[nickatnceas]

Initial rsyncs are running for cn-orc-1 and mn-orc-1. They're running pretty slow, at 20-30 MB/sec (each), but should finish tomorrow before noon PT if they keep that speed.

[nickatnceas]

cn-orc-1 and search-orc-1 are ready for final migrations. mn-orc-1 is still running its initial rsync.

[nickatnceas]

@taojing2002 @datadavev the initial rsyncs are done for cn-orc-1, mn-orc-1, and search-orc-1. Do you want to plan on doing the final migration tomorrow (Friday 10/15)?

I need to do the migrations one VM at a time. It will look something like:

1. Jing/Dave: Stop as many services as possible on the ORC VM (ie PG, Apache, Tomcat, etc)
2. Nick: Run a final rsync to UCSB
3. Nick: Update networking/grub/fstab/etc on the UCSB VM
4. Nick: Boot the UCSB VM
5. Nick: Change DNS
6. Jing/Dave: Check and fix DataONE services

I have started on cn-stage-orc-1 and cn-sandbox-orc-1 but they are not ready yet.

[taojing2002]

I just talked with Nick. We will first complete the sync for cn-sandbox/stage-orc-1 for testing, then do the final sync for production servers.

[nickatnceas]

The initial rsyncs for cn-stage-orc-1 and cn-sandbox-orc-1 are now running. They are transferring at speeds between 15-30 MB/sec, and have 1.1/1.2 TB to transfer, which will take about 22 hours at the slower 15 MB/sec speed.

We can cut the transfer time in half by deleting /var/postgres-bak on both VMs, but since it's running over the weekend, I don't think it's going to matter. Once the initial transfer is done subsequent rsyncs won't be affected much by those backup files.

I'm planning to be out on Monday Oct 18, but will be back Tuesday the 19th to do the final migrations.

[taojing2002]

Thanks, Nick! Please let me know before the final migration so I can shut down all services there.

…

On 10/15/21 3:57 PM, Nick Outin wrote: The initial rsyncs for cn-stage-orc-1 and cn-sandbox-orc-1 are now running. They are transferring at speeds between 15-30 MB/sec, and have 1.1/1.2 TB to transfer, which will take about 22 hours at the slower 15 MB/sec speed. We can cut the transfer time in half by deleting |/var/postgres-bak| on both VMs, but since it's running over the weekend, I don't think it's going to matter. Once the initial transfer is done subsequent rsyncs won't be affected much by those backup files. I'm planning to be out on Monday Oct 18, but will be back Tuesday the 19th to do the final migrations. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#13 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB5QQDEIC6AZUPQBUI73NJTUHCWXJANCNFSM5FUIYRAA>. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.