How to use weave method in pytest-benchmark
Best Python code snippet using pytest-benchmark
check_availability.py
Source:check_availability.py 
1CheckFeaturizer = {2 ('bace_c', 'logreg'): ['ECFP', 1024],3 ('bace_c', 'tf'): ['ECFP', 1024],4 ('bace_c', 'tf_robust'): ['ECFP', 1024],5 ('bace_c', 'rf'): ['ECFP', 1024],6 ('bace_c', 'kernelsvm'): ['ECFP', 1024],7 ('bace_c', 'irv'): ['ECFP', 1024],8 ('bace_c', 'xgb'): ['ECFP', 1024],9 ('bace_c', 'graphconv'): ['GraphConv', 75],10 ('bace_c', 'dag'): ['GraphConv', 75],11 ('bace_c', 'weave'): ['Weave', 75],12 ('bace_c', 'textcnn'): ['Raw', None],13 ('bace_c', 'mpnn'): ['Weave', [75, 14]],14 ('bbbp', 'logreg'): ['ECFP', 1024],15 ('bbbp', 'tf'): ['ECFP', 1024],16 ('bbbp', 'tf_robust'): ['ECFP', 1024],17 ('bbbp', 'rf'): ['ECFP', 1024],18 ('bbbp', 'kernelsvm'): ['ECFP', 1024],19 ('bbbp', 'irv'): ['ECFP', 1024],20 ('bbbp', 'xgb'): ['ECFP', 1024],21 ('bbbp', 'graphconv'): ['GraphConv', 75],22 ('bbbp', 'dag'): ['GraphConv', 75],23 ('bbbp', 'weave'): ['Weave', 75],24 ('bbbp', 'textcnn'): ['Raw', None],25 ('bbbp', 'mpnn'): ['Weave', [75, 14]],26 ('clintox', 'logreg'): ['ECFP', 1024],27 ('clintox', 'tf'): ['ECFP', 1024],28 ('clintox', 'tf_robust'): ['ECFP', 1024],29 ('clintox', 'rf'): ['ECFP', 1024],30 ('clintox', 'kernelsvm'): ['ECFP', 1024],31 ('clintox', 'irv'): ['ECFP', 1024],32 ('clintox', 'xgb'): ['ECFP', 1024],33 ('clintox', 'graphconv'): ['GraphConv', 75],34 ('clintox', 'dag'): ['GraphConv', 75],35 ('clintox', 'weave'): ['Weave', 75],36 ('clintox', 'textcnn'): ['Raw', None],37 ('clintox', 'mpnn'): ['Weave', [75, 14]],38 ('hiv', 'logreg'): ['ECFP', 1024],39 ('hiv', 'tf'): ['ECFP', 1024],40 ('hiv', 'tf_robust'): ['ECFP', 1024],41 ('hiv', 'rf'): ['ECFP', 1024],42 ('hiv', 'kernelsvm'): ['ECFP', 1024],43 ('hiv', 'irv'): ['ECFP', 1024],44 ('hiv', 'xgb'): ['ECFP', 1024],45 ('hiv', 'graphconv'): ['GraphConv', 75],46 ('hiv', 'dag'): ['GraphConv', 75],47 ('hiv', 'weave'): ['Weave', 75],48 ('hiv', 'textcnn'): ['Raw', None],49 ('hiv', 'mpnn'): ['Weave', [75, 14]],50 ('muv', 'logreg'): ['ECFP', 1024],51 ('muv', 'tf'): ['ECFP', 1024],52 ('muv', 'tf_robust'): ['ECFP', 1024],53 ('muv', 'rf'): ['ECFP', 1024],54 ('muv', 'kernelsvm'): ['ECFP', 1024],55 ('muv', 'irv'): ['ECFP', 1024],56 ('muv', 'xgb'): ['ECFP', 1024],57 ('muv', 'graphconv'): ['GraphConv', 75],58 ('muv', 'siamese'): ['GraphConv', 75],59 ('muv', 'attn'): ['GraphConv', 75],60 ('muv', 'res'): ['GraphConv', 75],61 ('muv', 'weave'): ['Weave', 75],62 ('muv', 'textcnn'): ['Raw', None],63 ('muv', 'mpnn'): ['Weave', [75, 14]],64 ('pcba', 'logreg'): ['ECFP', 1024],65 ('pcba', 'tf'): ['ECFP', 1024],66 ('pcba', 'tf_robust'): ['ECFP', 1024],67 ('pcba', 'irv'): ['ECFP', 1024],68 ('pcba', 'xgb'): ['ECFP', 1024],69 ('pcba', 'graphconv'): ['GraphConv', 75],70 ('pcba', 'weave'): ['Weave', 75],71 ('pcba', 'textcnn'): ['Raw', None],72 ('pcba_146', 'logreg'): ['ECFP', 1024],73 ('pcba_146', 'tf'): ['ECFP', 1024],74 ('pcba_146', 'tf_robust'): ['ECFP', 1024],75 ('pcba_146', 'irv'): ['ECFP', 1024],76 ('pcba_146', 'xgb'): ['ECFP', 1024],77 ('pcba_146', 'graphconv'): ['GraphConv', 75],78 ('pcba_146', 'weave'): ['Weave', 75],79 ('pcba_2475', 'logreg'): ['ECFP', 1024],80 ('pcba_2475', 'tf'): ['ECFP', 1024],81 ('pcba_2475', 'tf_robust'): ['ECFP', 1024],82 ('pcba_2475', 'irv'): ['ECFP', 1024],83 ('pcba_2475', 'xgb'): ['ECFP', 1024],84 ('pcba_2475', 'graphconv'): ['GraphConv', 75],85 ('pcba_2475', 'weave'): ['Weave', 75],86 ('sider', 'logreg'): ['ECFP', 1024],87 ('sider', 'tf'): ['ECFP', 1024],88 ('sider', 'tf_robust'): ['ECFP', 1024],89 ('sider', 'rf'): ['ECFP', 1024],90 ('sider', 'kernelsvm'): ['ECFP', 1024],91 ('sider', 'irv'): ['ECFP', 1024],92 ('sider', 'xgb'): ['ECFP', 1024],93 ('sider', 'graphconv'): ['GraphConv', 75],94 ('sider', 'dag'): ['GraphConv', 75],95 ('sider', 'weave'): ['Weave', 75],96 ('sider', 'siamese'): ['GraphConv', 75],97 ('sider', 'attn'): ['GraphConv', 75],98 ('sider', 'res'): ['GraphConv', 75],99 ('sider', 'textcnn'): ['Raw', None],100 ('sider', 'mpnn'): ['Weave', [75, 14]],101 ('tox21', 'logreg'): ['ECFP', 1024],102 ('tox21', 'tf'): ['ECFP', 1024],103 ('tox21', 'tf_robust'): ['ECFP', 1024],104 ('tox21', 'rf'): ['ECFP', 1024],105 ('tox21', 'kernelsvm'): ['ECFP', 1024],106 ('tox21', 'irv'): ['ECFP', 1024],107 ('tox21', 'xgb'): ['ECFP', 1024],108 ('tox21', 'graphconv'): ['GraphConv', 75],109 ('tox21', 'dag'): ['GraphConv', 75],110 ('tox21', 'weave'): ['Weave', 75],111 ('tox21', 'siamese'): ['GraphConv', 75],112 ('tox21', 'attn'): ['GraphConv', 75],113 ('tox21', 'res'): ['GraphConv', 75],114 ('tox21', 'textcnn'): ['Raw', None],115 ('tox21', 'mpnn'): ['Weave', [75, 14]],116 ('toxcast', 'logreg'): ['ECFP', 1024],117 ('toxcast', 'tf'): ['ECFP', 1024],118 ('toxcast', 'tf_robust'): ['ECFP', 1024],119 ('toxcast', 'rf'): ['ECFP', 1024],120 ('toxcast', 'kernelsvm'): ['ECFP', 1024],121 ('toxcast', 'irv'): ['ECFP', 1024],122 ('toxcast', 'xgb'): ['ECFP', 1024],123 ('toxcast', 'graphconv'): ['GraphConv', 75],124 ('toxcast', 'weave'): ['Weave', 75],125 ('toxcast', 'textcnn'): ['Raw', None],126 ('toxcast', 'mpnn'): ['Weave', [75, 14]],127 ('bace_r', 'tf_regression'): ['ECFP', 1024],128 ('bace_r', 'rf_regression'): ['ECFP', 1024],129 ('bace_r', 'krr'): ['ECFP', 1024],130 ('bace_r', 'xgb_regression'): ['ECFP', 1024],131 ('bace_r', 'graphconvreg'): ['GraphConv', 75],132 ('bace_r', 'dag_regression'): ['GraphConv', 75],133 ('bace_r', 'weave_regression'): ['Weave', 75],134 ('bace_r', 'textcnn_regression'): ['Raw', None],135 ('chembl', 'tf_regression'): ['ECFP', 1024],136 ('chembl', 'rf_regression'): ['ECFP', 1024],137 ('chembl', 'krr'): ['ECFP', 1024],138 ('chembl', 'xgb_regression'): ['ECFP', 1024],139 ('chembl', 'graphconvreg'): ['GraphConv', 75],140 ('chembl', 'weave_regression'): ['Weave', 75],141 ('clearance', 'tf_regression'): ['ECFP', 1024],142 ('clearance', 'rf_regression'): ['ECFP', 1024],143 ('clearance', 'krr'): ['ECFP', 1024],144 ('clearance', 'xgb_regression'): ['ECFP', 1024],145 ('clearance', 'graphconvreg'): ['GraphConv', 75],146 ('clearance', 'dag_regression'): ['GraphConv', 75],147 ('clearance', 'weave_regression'): ['Weave', 75],148 ('delaney', 'tf_regression'): ['ECFP', 1024],149 ('delaney', 'rf_regression'): ['ECFP', 1024],150 ('delaney', 'krr'): ['ECFP', 1024],151 ('delaney', 'xgb_regression'): ['ECFP', 1024],152 ('delaney', 'graphconvreg'): ['GraphConv', 75],153 ('delaney', 'dag_regression'): ['GraphConv', 75],154 ('delaney', 'weave_regression'): ['Weave', 75],155 ('delaney', 'mpnn'): ['Weave', [75, 14]],156 ('delaney', 'textcnn_regression'): ['Raw', None],157 ('hopv', 'tf_regression'): ['ECFP', 1024],158 ('hopv', 'rf_regression'): ['ECFP', 1024],159 ('hopv', 'krr'): ['ECFP', 1024],160 ('hopv', 'xgb_regression'): ['ECFP', 1024],161 ('hopv', 'graphconvreg'): ['GraphConv', 75],162 ('hopv', 'dag_regression'): ['GraphConv', 75],163 ('hopv', 'weave_regression'): ['Weave', 75],164 ('lipo', 'tf_regression'): ['ECFP', 1024],165 ('lipo', 'rf_regression'): ['ECFP', 1024],166 ('lipo', 'krr'): ['ECFP', 1024],167 ('lipo', 'xgb_regression'): ['ECFP', 1024],168 ('lipo', 'graphconvreg'): ['GraphConv', 75],169 ('lipo', 'dag_regression'): ['GraphConv', 75],170 ('lipo', 'weave_regression'): ['Weave', 75],171 ('lipo', 'mpnn'): ['Weave', [75, 14]],172 ('lipo', 'textcnn_regression'): ['Raw', None],173 ('nci', 'tf_regression'): ['ECFP', 1024],174 ('nci', 'rf_regression'): ['ECFP', 1024],175 ('nci', 'krr'): ['ECFP', 1024],176 ('nci', 'xgb_regression'): ['ECFP', 1024],177 ('nci', 'graphconvreg'): ['GraphConv', 75],178 ('nci', 'weave_regression'): ['Weave', 75],179 ('ppb', 'tf_regression'): ['ECFP', 1024],180 ('ppb', 'rf_regression'): ['ECFP', 1024],181 ('ppb', 'krr'): ['ECFP', 1024],182 ('ppb', 'xgb_regression'): ['ECFP', 1024],183 ('ppb', 'graphconvreg'): ['GraphConv', 75],184 ('ppb', 'dag_regression'): ['GraphConv', 75],185 ('ppb', 'weave_regression'): ['Weave', 75],186 ('sampl', 'tf_regression'): ['ECFP', 1024],187 ('sampl', 'rf_regression'): ['ECFP', 1024],188 ('sampl', 'krr'): ['ECFP', 1024],189 ('sampl', 'xgb_regression'): ['ECFP', 1024],190 ('sampl', 'graphconvreg'): ['GraphConv', 75],191 ('sampl', 'dag_regression'): ['GraphConv', 75],192 ('sampl', 'weave_regression'): ['Weave', 75],193 ('sampl', 'mpnn'): ['Weave', [75, 14]],194 ('sampl', 'textcnn_regression'): ['Raw', None],195 ## added on 2/17/2019196 ('flashpoint', 'tf_regression'): ['ECFP', 1024],197 ('flashpoint', 'rf_regression'): ['ECFP', 1024],198 ('flashpoint', 'krr'): ['ECFP', 1024],199 ('flashpoint', 'xgb_regression'): ['ECFP', 1024],200 ('flashpoint', 'graphconvreg'): ['GraphConv', 75],201 ('flashpoint', 'dag_regression'): ['GraphConv', 75],202 ('flashpoint', 'weave_regression'): ['Weave', 75],203 ('flashpoint', 'mpnn'): ['Weave', [75, 14]],204 ('flashpoint', 'textcnn_regression'): ['Raw', None],205 ## added on 2/17/2019206 ('kaggle', 'tf_regression'): [None, 14293],207 ('kaggle', 'rf_regression'): [None, 14293],208 ('kaggle', 'krr'): [None, 14293],209 ('pdbbind', 'tf_regression'): ['grid', 2052],210 ('pdbbind', 'rf_regression'): ['grid', 2052],211 ('pdbbind', 'krr'): ['grid', 2052],212 ('pdbbind', 'graphconvreg'): ['GraphConv', 75],213 ('qm7', 'tf_regression'): ['ECFP', 1024],214 ('qm7', 'rf_regression'): ['ECFP', 1024],215 ('qm7', 'krr'): ['ECFP', 1024],216 ('qm7', 'krr_ft'): ['CoulombMatrix', 1024],217 ('qm7', 'textcnn_regression'): ['Raw', None],218 ('qm7', 'graphconvreg'): ['GraphConv', 75],219 ('qm7', 'weave_regression'): ['Weave', 75],220 ('qm7', 'tf_regression_ft'): ['CoulombMatrix', [23, 23]],221 ('qm7', 'dtnn'): ['CoulombMatrix', [23, 23]],222 ('qm7', 'ani'): ['BPSymmetryFunction', [23, 4]],223 ('qm7b', 'tf_regression_ft'): ['CoulombMatrix', [23, 23]],224 ('qm7b', 'krr_ft'): ['CoulombMatrix', 1024],225 ('qm7b', 'dtnn'): ['CoulombMatrix', [23, 23]],226 ('qm8', 'tf_regression'): ['ECFP', 1024],227 ('qm8', 'rf_regression'): ['ECFP', 1024],228 ('qm8', 'krr'): ['ECFP', 1024],229 ('qm8', 'graphconvreg'): ['GraphConv', 75],230 ('qm8', 'tf_regression_ft'): ['CoulombMatrix', [26, 26]],231 ('qm8', 'krr_ft'): ['CoulombMatrix', 1024],232 ('qm8', 'dtnn'): ['CoulombMatrix', [26, 26]],233 ('qm8', 'ani'): ['BPSymmetryFunction', [26, 4]],234 ('qm8', 'mpnn'): ['MP', [70, 8]],235 ('qm8', 'weave_regression'): ['Weave', 75],236 ('qm8', 'textcnn_regression'): ['Raw', None],237 ('qm9', 'tf_regression'): ['ECFP', 1024],238 ('qm9', 'rf_regression'): ['ECFP', 1024],239 ('qm9', 'krr'): ['ECFP', 1024],240 ('qm9', 'graphconvreg'): ['GraphConv', 75],241 ('qm9', 'tf_regression_ft'): ['CoulombMatrix', [29, 29]],242 ('qm9', 'krr_ft'): ['CoulombMatrix', 1024],243 ('qm9', 'dtnn'): ['CoulombMatrix', [29, 29]],244 ('qm9', 'ani'): ['BPSymmetryFunction', [29, 4]],245 ('qm9', 'mpnn'): ['MP', [70, 8]],246 ('qm9', 'weave_regression'): ['Weave', 75],247 ('qm9', 'textcnn_regression'): ['Raw', None]248}249CheckSplit = {250 'bace_c': ['random', 'scaffold'],251 'bace_r': ['random', 'scaffold'],252 'bbbp': ['random', 'scaffold'],253 'chembl': ['index', 'random', 'scaffold', 'year'],254 'clearance': ['index', 'random', 'scaffold'],255 'clintox': ['index', 'random', 'scaffold'],256 'delaney': ['index', 'random', 'scaffold'],257 'hiv': ['index', 'random', 'scaffold', 'butina'],258 'hopv': ['index', 'random', 'scaffold', 'butina'],259 'kaggle': ['index'], # already splitted, no splitter required260 'lipo': ['index', 'random', 'scaffold'],261 'muv': ['index', 'random', 'scaffold', 'task'],262 'nci': ['index', 'random', 'scaffold'],263 'pcba': ['index', 'random', 'scaffold'],264 'pcba_146': ['index', 'random', 'scaffold'],265 'pcba_2475': ['index', 'random', 'scaffold'],266 'pdbbind': ['index', 'random', 'time'],267 'ppb': ['index', 'random', 'scaffold'],268 'qm7': ['index', 'random', 'stratified'],269 'qm7b': ['index', 'random', 'stratified'],270 'qm8': ['index', 'random', 'stratified'],271 'qm9': ['index', 'random', 'stratified'],272 'sampl': ['index', 'random', 'scaffold'],273 'flashpoint': ['index', 'random', 'scaffold'], # added on 2/17/2019 by Xiaoyu Sun274 'sider': ['index', 'random', 'scaffold', 'task'],275 'tox21': ['index', 'random', 'scaffold', 'butina', 'task'],276 'toxcast': ['index', 'random', 'scaffold']...
vm.py
Source:vm.py
1"""2Licensed to the Apache Software Foundation (ASF) under one3or more contributor license agreements. See the NOTICE file4distributed with this work for additional information5regarding copyright ownership. The ASF licenses this file6to you under the Apache License, Version 2.0 (the7"License"); you may not use this file except in compliance8with the License. You may obtain a copy of the License at9 http://www.apache.org/licenses/LICENSE-2.010Unless required by applicable law or agreed to in writing, software11distributed under the License is distributed on an "AS IS" BASIS,12WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.13See the License for the specific language governing permissions and14limitations under the License.15"""16import subprocess17import shutil18from config import Config19import os20from docker import Docker21from docker_image.launcher_agent import replace_conf22class VM:23 """24 This class represents VM, including its network setting and the possible Docker instance list25 """26 def __init__(self, external_ip, domain_name, weave_dns_ip, weave_ip_mask):27 self.external_ip = external_ip28 self.domain_name = domain_name29 self.hostname = self._gce_get_hostname(domain_name)30 self.weave_domain_name = self._get_weave_domain_name(self.hostname)31 self.weave_dns_ip = weave_dns_ip32 self.weave_internal_ip = ""33 self.weave_ip_mask = weave_ip_mask34 self.docker_list = []35 def to_json(self):36 """37 create a map to hold the information of the VM instance38 :return: A map, which is JSON format object.39 """40 vm_json = {}41 vm_json["external_ip"] = self.external_ip42 vm_json["domain_name"] = self.domain_name43 vm_json["weave_dns_ip"] = self.weave_dns_ip44 vm_json["weave_internal_ip"] = self.weave_internal_ip45 vm_json["weave_domain_name"] = self.weave_domain_name46 vm_json["weave_ip_mask"] = self.weave_ip_mask47 vm_json["docker_list"] = []48 for docker in self.docker_list:49 vm_json["docker_list"].append(docker.to_json())50 return vm_json51 @staticmethod52 def load_from_json(json_data):53 """54 load the VM information from a JSON object55 :param json_data: a map, which is a JSON object56 :return: a VM object57 """58 external_ip = json_data["external_ip"]59 domain_name = json_data["domain_name"]60 weave_dns_ip = json_data["weave_dns_ip"]61 weave_internal_ip = json_data["weave_internal_ip"]62 weave_domain_name = json_data["weave_domain_name"]63 weave_ip_mask = json_data["weave_ip_mask"]64 docker_list = []65 for json_docker in json_data["docker_list"]:66 docker_list.append(Docker.load_from_json(json_docker))67 vm = VM(external_ip, domain_name, weave_dns_ip, weave_ip_mask)68 vm.docker_list = docker_list69 vm.weave_internal_ip = weave_internal_ip70 vm.weave_domain_name = weave_domain_name71 return vm72 def _get_weave_domain_name(self, hostname):73 """74 get the Weave domain name of the VM75 :param hostname: the hostname of the VM76 :return:the Weave domain name77 """78 return "{0}.weave.local".format(hostname)79 def _gce_get_hostname(self, domain_name):80 """81 The hostname of GCE VM is the first part of the internal domain name82 :param domain_name: the internal domain name of GCE VM83 :return: the hostname of GCE VM84 """85 return domain_name.split(".")[0]86 def get_ssh_output_file_path(self):87 """88 get the file name to hold the SSH output of the VM89 :return: a file name90 """91 vm_output_file_path = "{0}/vm-{1}-{2}".format(Config.ATTRIBUTES["output_folder"],92 self.hostname, self.external_ip)93 return vm_output_file_path94 def add_docker(self, docker):95 """96 add a Docker instance to the VM instance97 :param docker: the docker instance98 :return: None99 """100 self.docker_list.append(docker)101 def _centos7_weave_install(self):102 """103 install Weave on this VM104 :return: None105 """106 subprocess.call("./Linux/CentOS7/weave_install.sh")107 def _set_weave_network(self, vm_external_ip_list, weave_dns_ip):108 """109 launch Weave, make this VM connect with other VM110 :param vm_external_ip_list: external IP list of all VMs111 :param weave_dns_ip: the IP of DNS in this VM112 :return: None113 """114 # add other VMs and the ambari-server to set up connections115 weave_launch_command = ["sudo", "weave", "launch"]116 weave_launch_command.extend(vm_external_ip_list)117 print weave_launch_command118 with open(os.devnull, 'w') as shutup:119 subprocess.call(weave_launch_command, stdout=shutup)120 # establish DNS server121 weave_dns_ip_with_mask = "{0}/{1}".format(weave_dns_ip, Config.ATTRIBUTES["weave_ip_mask"])122 weave_launch_dns_command = ["sudo", "weave", "launch-dns", weave_dns_ip_with_mask]123 subprocess.call(weave_launch_dns_command)124 def _centos7_docker_install(self):125 """126 install Docker on this VM127 :return: None128 """129 subprocess.call("./Linux/CentOS7/docker_install.sh")130 def _build_docker_image(self, image_name):131 """132 build docker image133 :param image_name: the name of the Docker image134 :return: None135 """136 # choose the right Dockerfile137 target_dockerfile_name = "docker_image/{0}".format(Config.ATTRIBUTES["dockerfile_name"])138 standard_dockerfile_name = "docker_image/Dockerfile"139 shutil.copyfile(target_dockerfile_name, standard_dockerfile_name)140 with open(os.devnull, 'w') as shutup:141 subprocess.call(["sudo", "docker", "build", "-t", image_name, "docker_image/"])142 # subprocess.call(["sudo", "docker", "build", "-q", "-t", image_name, "docker_image/"], stdout=shutup)143 os.remove(standard_dockerfile_name)144 def _pull_docker_image(self, image_name):145 with open(os.devnull, 'w') as shutup:146 subprocess.call(["sudo", "docker", "pull", image_name], stdout=shutup)147 def _launch_containers(self, docker_image, server_weave_ip):148 """149 launch Docker containers, issue the script to install,150 configure and launch Ambari-gent inside Docker.151 :param docker_image: the name of the Docker image152 :param server_weave_ip: Weave internal IP of Ambari-server153 :return: None154 """155 for docker in self.docker_list:156 docker_ip_with_mask = "{0}/{1}".format(docker.ip, docker.mask)157 cmd = "python /launcher_agent.py {0} {1}; /bin/bash".format(server_weave_ip, docker.ip)158 command = ["sudo", "weave", "run", docker_ip_with_mask, "-d", "-it",159 "-h", docker.weave_domain_name,160 "--name", docker.get_container_name(),161 docker_image, "bash", "-c", cmd]162 print command163 subprocess.call(command)164 def _set_docker_partition(self, mount_point):165 """166 set docker container to use the disk storage of other partitions.167 :param mount_point: the mount point of the partition to be used168 :return: None169 """170 subprocess.call(["./Linux/CentOS7/set_docker_partition.sh", mount_point])171 def run_ambari_server(self):172 """173 set up Weave network, run Ambari-server in this VM174 :return: None175 """176 # set up network, run script inside the network directory177 os.chdir("network")178 subprocess.call(["./set_ambari_server_network.sh", self.weave_internal_ip,179 self.weave_dns_ip, self.weave_ip_mask])180 os.chdir("..")181 # install ambari server and start service182 subprocess.call(["./server/ambari_server_install.sh"])183 # start service184 subprocess.call(["./server/ambari_server_start.sh"])185 def run_service_server(self, ambari_server_weave_ip, ambari_server_external_ip):186 """187 set up Weave network, run Ambari-agent in this VM188 :param ambari_server_weave_ip: the Weave IP of Ambari-server189 :param ambari_server_external_ip: the external IP of Ambari-server190 :return: None191 """192 # set up network, run script inside the network directory193 os.chdir("network")194 subprocess.call(["./set_host_network.sh", self.weave_internal_ip,195 self.weave_dns_ip, self.weave_ip_mask, self.hostname,196 self.weave_domain_name, ambari_server_external_ip])197 os.chdir("..")198 # install ambari agent and start service199 subprocess.call(["./docker_image/ambari_agent_install.sh"])200 replace_conf(ambari_server_weave_ip)201 # start service202 subprocess.call(["./docker_image/ambari_agent_start.sh"])203 # forward public IP to Weave IP for server UI access204 port_list = Config.ATTRIBUTES["server_port_list"].split(",")205 for port in port_list:206 subprocess.call(["./network/set_ui_port_forward.sh", self.external_ip, self.weave_internal_ip, port])207 def run_docker(self, server_weave_ip, vm_ip_list):208 """209 run all Docker containers with Ambari-agent inside210 :param server_weave_ip: Weave internal IP of Ambari-server211 :param vm_ip_list: external IP list of all other VMs to be connected212 each docker vm connect to each other and service VM (and ambari-server)213 :return: None214 """215 self._centos7_docker_install()216 if "use_partition" in Config.ATTRIBUTES:217 self._set_docker_partition(Config.ATTRIBUTES["use_partition"])218 self._centos7_weave_install()219 image_name = Config.ATTRIBUTES["docker_image_name"]220 if "pull_docker_hub" in Config.ATTRIBUTES and Config.ATTRIBUTES["pull_docker_hub"] == "yes":221 self._pull_docker_image(image_name)222 else:223 self._build_docker_image(image_name)224 self._set_weave_network(vm_ip_list, self.weave_dns_ip)225 self._launch_containers(Config.ATTRIBUTES["docker_image_name"], server_weave_ip)226 @staticmethod227 def get_ambari_agent_vm_name(cluster_name):228 return "{0}-agent-vm".format(cluster_name)229 @staticmethod230 def get_ambari_server_vm_name(cluster_name):231 return "{0}-ambari-server".format(cluster_name)232 @staticmethod233 def get_service_server_vm_name(cluster_name):...
docker.py
Source:docker.py
1"""2Licensed to the Apache Software Foundation (ASF) under one3or more contributor license agreements. See the NOTICE file4distributed with this work for additional information5regarding copyright ownership. The ASF licenses this file6to you under the Apache License, Version 2.0 (the7"License"); you may not use this file except in compliance8with the License. You may obtain a copy of the License at9 http://www.apache.org/licenses/LICENSE-2.010Unless required by applicable law or agreed to in writing, software11distributed under the License is distributed on an "AS IS" BASIS,12WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.13See the License for the specific language governing permissions and14limitations under the License.15"""16from config import Config17class Docker:18 """19 Docker represents a Docker container, each with its IP and domain name20 """21 def __init__(self, ip, mask, weave_domain_name):22 self.ip = ip23 self.mask = mask24 self.weave_domain_name = weave_domain_name25 def to_json(self):26 """27 create a map to hold the information of the Docker instance28 :return: A map, which is JSON format object.29 """30 docker_json = {}31 docker_json["weave_ip"] = "{0}/{1}".format(self.ip, self.mask)32 docker_json["weave_domain_name"] = self.weave_domain_name33 return docker_json34 @staticmethod35 def load_from_json(json_data):36 """37 load the docker information from a JSON object38 :param json_data: a map, which is a JSON object39 :return: a Docker object40 """41 ip = json_data["weave_ip"].split("/")[0]42 mask = json_data["weave_ip"].split("/")[1]43 weave_domain_name = json_data["weave_domain_name"]44 return Docker(ip, mask, weave_domain_name)45 def __str__(self):46 return str(self.ip) + "/" + str(self.mask) + " " + self.weave_domain_name47 @staticmethod48 def get_weave_domain_name(cluster_name, index):49 """50 given the index and the name of cluster, generate the Weave domain name for the docker51 :param cluster_name: the name of the cluster52 :param index: a number53 :return: Weave domain name of the docker container54 """55 return "{0}-{1}-{2}.{3}".format(Config.ATTRIBUTES["container_hostname_fix"],56 index, cluster_name, "weave.local")57 @staticmethod58 def get_pattern_presentation(cluster_name, range_str):59 return Docker.get_weave_domain_name(cluster_name, range_str)60 def get_index(self):61 """62 extract the index of the docker within the cluster63 :return: the index64 """65 return self.weave_domain_name.split("-")[1]66 def get_container_name(self):67 """68 :return: the name of the container69 """70 return self.get_hostname()71 def get_hostname(self):...
Automation Testing Tutorials
Learn to execute automation testing from scratch with LambdaTest Learning Hub. Right from setting up the prerequisites to run your first automation test, to following best practices and diving deeper into advanced test scenarios. LambdaTest Learning Hubs compile a list of step-by-step guides to help you be proficient with different test automation frameworks i.e. Selenium, Cypress, TestNG etc.
LambdaTest Learning Hubs:
- JUnit Tutorial
- TestNG Tutorial
- Webdriver Tutorial
- WebDriverIO Tutorial
- Protractor Tutorial
- Selenium 4 Tutorial
- Jenkins Tutorial
- NUnit Tutorial
- Jest Tutorial
- Playwright Tutorial
- Cypress Tutorial
- PyTest Tutorial
YouTube
You could also refer to video tutorials over LambdaTest YouTube channel to get step by step demonstration from industry experts.
Run pytest-benchmark automation tests on LambdaTest cloud grid
Perform automation testing on 3000+ real desktop and mobile devices online.
Try LambdaTest Now !!
Get 100 minutes of automation test minutes FREE!!
Was this article helpful?
