Unlock 30% off on Manual Testing Annual Plans this Holiday Season. 
Use Code: LCMYEARENDEXCL30
Use Code: LCMYEARENDEXCL30Check icon
Copied to Clipboard!
How to use test_all method in autotest
Best Python code snippet using autotest_python
Text Mining Script.py
Source:Text Mining Script.py 
1# -*- coding: utf-8 -*-2"""3Created on Sat Mar 19 09:13:22 201645@author: Team A-066"""7import string8import numpy as np9import pandas as pd10import random11import matplotlib.pyplot as plt12random.seed(120)13import os14os.getcwd()1516Train = pd.read_csv('train.csv',encoding ="ISO-8859-1")17Test = pd.read_csv('test.csv',encoding ="ISO-8859-1")18len(Train)19Train_number = Train.shape[0]2021Prod_Desc = pd.read_csv('product_descriptions.csv',encoding ="ISO-8859-1")22Prod_Attr = pd.read_csv('attributes.csv',encoding ="ISO-8859-1")23Prod_Attr.head()2425Prod_attrBrandname = Prod_Attr[Prod_Attr.name== "MFG Brand Name"][["product_uid","value"]].rename(columns={"value":"Brand"})26len(Prod_attrBrandname)2728# Add features to the Train set from product description file and the attributes file29Train_all = pd.merge(Train,Prod_Desc,on='product_uid',how='left')30Train_all = pd.merge(Train_all,Prod_attrBrandname,on='product_uid',how='left')3132#Replacing missing values with Mode33Train_all['Brand'] = np.where(Train_all.Brand.isnull(),Train_all.Brand.mode(),Train_all.Brand)34Train_all.info()3536# Add features to the Test set37Test_all = pd.merge(Test,Prod_Desc,on='product_uid',how='left')38Test_all = pd.merge(Test_all,Prod_attrBrandname,on='product_uid',how='left')39#Replacing missing values with Mode40Test_all['Brand'] = np.where(Test_all.Brand.isnull(),Test_all.Brand.mode(),Test_all.Brand)4142Test_all.info()4344Y_train = Train_all['relevance']4546#Parsing Text Columns47from datetime import datetime48start = datetime.now()49# Preprocessing text data50#from nltk.corpus import stopwords51Num = {'zero':0,'one':1,'two':2,'three':3,'four':4,'five':5,'six':6,'seven':7,'eight':8,'nine':9}52from stemming.porter2 import stem53import re54#from textblob import TextBlob55def textprocess(term):56 if isinstance(term,str):57 term = re.sub(r"([0-9]+)( *)(inches|inch|in|')\.?", r"\inches ", term)58 term = re.sub(r"([0-9]+)( *)(foot|feet|ft|'')\.?", r"\feet ", term)59 term = re.sub(r"([0-9]+)( *)(pounds|pound|lbs|lb)\.?", r"\lb ", term)60 term = re.sub(r"([0-9]+)( *)(square|sq) ?\.?(feet|foot|ft)\.?", r"\1sq.ft. ", term)61 term = re.sub(r"([0-9]+)( *)(cubic|cu) ?\.?(feet|foot|ft)\.?", r"\1cu.ft. ", term)62 term = re.sub(r"([0-9]+)( *)(gallons|gallon|gal)\.?", r"\1gal. ", term)63 term = re.sub(r"([0-9]+)( *)(ounces|ounce|oz)\.?", r"\1oz. ", term)64 term = re.sub(r"([0-9]+)( *)(centimeters|cm)\.?", r"\1cm. ", term)65 term = re.sub(r"([0-9]+)( *)(milimeters|mm)\.?", r"\1mm. ", term)66 term = term.replace("°"," degrees ")67 term = re.sub(r"([0-9]+)( *)(degrees|degree)\.?", r"\1deg. ", term)68 term = term.replace(" v "," volts ")69 term = re.sub(r"([0-9]+)( *)(volts|volt)\.?", r"\1volt. ", term)70 term = re.sub(r"([0-9]+)( *)(watts|watt)\.?", r"\1watt. ", term)71 term = re.sub(r"([0-9]+)( *)(amperes|ampere|amps|amp)\.?", r"\1amp. ", term)72 term = term.replace(" "," ")73 term = term.replace(" . "," ")74 term = (" ").join([str(Num[z]) if z in Num else z for z in term.split(" ")])75 term = term.replace("toliet","toilet")76 term = term.replace("airconditioner","air conditioner")77 term = term.replace("vinal","vinyl")78 term = term.replace("vynal","vinyl")79 term = term.replace("skill","skil")80 term = term.replace("snowbl","snow bl")81 term = term.replace("plexigla","plexi gla")82 term = term.replace("rustoleum","rust-oleum")83 term = term.replace("whirpool","whirlpool")84 term = term.replace("whirlpoolga", "whirlpool ga")85 term = term.replace("whirlpoolstainless","whirlpool stainless")86 #Perform Stemming87 stemtxt = [stem(word) for word in term]88 lowertxt = [word.lower() for word in stemtxt]89 90 # Remove Punctuation91 nopunc = [word for word in lowertxt if word not in string.punctuation]92 nopunc = ''.join(nopunc)93 #Remove stopwords94# cleantxt = [word for word in nopunc.split() if word not in stopwords.words('english')]95# cleantxt = ''.join(cleantxt)96 return nopunc9798#Preprocessing of Train set99start = datetime.now() 100Train_all['product_title'] = Train_all['product_title'].map(lambda x:textprocess(x))101Train_all['search_term'] = Train_all['search_term'].map(lambda x:textprocess(x))102Train_all['product_description'] = Train_all['product_description'].map(lambda x:textprocess(x))103Train_all['Brand'] = Train_all['Brand'].map(lambda x:textprocess(x))104Train_all['prodtext'] = Train_all['search_term']+" "+Train_all['product_title']+" "+Train_all['product_description']+" "+Train_all['Brand']105Train_all['prodtext1'] = Train_all['search_term']+"\t"+Train_all['product_title']+"\t"+Train_all['Brand']+"\t"+Train_all['product_description']106print(datetime.now() - start)107108Train_all.prodtext.head()109110#Preprocessing for Test set111start = datetime.now() 112Test_all['product_title'] = Test_all['product_title'].map(lambda x:textprocess(x))113Test_all['search_term'] = Test_all['search_term'].map(lambda x:textprocess(x))114Test_all['product_description'] = Test_all['product_description'].map(lambda x:textprocess(x))115Test_all['Brand'] = Test_all['Brand'].map(lambda x:textprocess(x))116Test_all['prodtext'] = Test_all['search_term']+" "+Test_all['product_title']+" "+Test_all['Brand']+" "+Test_all['product_description']117Test_all['prodtext1'] = Test_all['search_term']+"\t"+Test_all['product_title']+"\t"+Test_all['Brand']+"\t"+Test_all['product_description']118print(datetime.now() - start)119Test_all.prodtext.head()120121Train_all.info()122Test_all.info()123Y_train = Train_all['relevance']124X_train = Train_all['prodtext']125X_test = Test_all['prodtext']126127128#Cross validation129start = datetime.now() 130131from sklearn.utils import shuffle132133X,y = shuffle(X_train,Y_train,random_state = 13)134offset = int(X.shape[0] *0.80)135X_crosstrain , y_crosstrain = X[:offset],y[:offset]136X_crosstest , y_crosstest = X[offset:],y[offset:]137138from sklearn.feature_extraction.text import TfidfVectorizer139tfidf = TfidfVectorizer(max_df=0.4, max_features=None,analyzer='char_wb',ngram_range=(1,2), 140 use_idf=True, smooth_idf=True, sublinear_tf=True, stop_words = 'english')141142143X_crosstrain = tfidf.fit_transform(X_crosstrain)144145X_crosstest = tfidf.transform(X_crosstest)146147from sklearn.decomposition import TruncatedSVD148from sklearn.preprocessing import StandardScaler149150svd = TruncatedSVD(n_components=300, algorithm='randomized', n_iter=5, random_state=None, tol=0.0)151X_crosstrain_svd = svd.fit_transform(X_crosstrain)152svd.explained_variance_ratio_.sum()153X_crosstest_svd = svd.transform(X_crosstest)154155scl = StandardScaler(copy=True, with_mean=True, with_std=True)156157X_crosstrain_scl = scl.fit_transform(X_crosstrain_svd)158X_crosstest_scl = scl.transform(X_crosstest_svd)159160from sklearn import ensemble161from sklearn.utils import shuffle162from sklearn.metrics import mean_squared_error163164start = datetime.now() 165from sklearn.svm import SVR166from sklearn.metrics import mean_squared_error167168Model_one = SVR(C=1.0, kernel='rbf', degree=3, gamma='auto', coef0=0.0, shrinking=True, 169 tol=0.001, cache_size=200,verbose=False, max_iter=30000)170Model_one.fit(X_crosstrain_scl,y_crosstrain)171SVMmse = Model_one.predict(X_crosstest_scl)172print("SVM_RMSE:",np.sqrt(mean_squared_error(y_crosstest,SVMmse)))173print(datetime.now() - start)174175start = datetime.now() 176from sklearn.neighbors import KNeighborsRegressor177Model_two = KNeighborsRegressor(n_neighbors=11)178Model_two.fit(X_crosstrain_scl,y_crosstrain)179KNNmse = Model_two.predict(X_crosstest_scl)180print("KNN_RMSE:",np.sqrt(mean_squared_error(y_crosstest,KNNmse)))181print(datetime.now() - start)182183start = datetime.now() 184from sklearn import ensemble185Model_three = ensemble.RandomForestRegressor(n_estimators = 500,verbose=1,n_jobs=-1,random_state = 120,max_depth=16)186Model_three.fit(X_crosstrain_svd,y_crosstrain)187RFmse = Model_three.predict(X_crosstest_svd)188print("RandomForest_RMSE:",np.sqrt(mean_squared_error(y_crosstest,RFmse)))189print(datetime.now() - start)190191start = datetime.now() 192from sklearn.linear_model import BayesianRidge193BR = BayesianRidge(n_iter=500,tol= 0.001,normalize=True).fit(X_crosstrain_scl,y_crosstrain)194pred_BR = BR.predict(X_crosstest_scl)195print("BayesinRidge_RMSE:",np.sqrt(mean_squared_error(y_crosstest,pred_BR)))196print(datetime.now() - start)197198start = datetime.now() 199from sklearn.linear_model import LinearRegression200LR = LinearRegression(fit_intercept = True,normalize = True,n_jobs=-1).fit(X_crosstrain_svd,y_crosstrain)201pred_LR = LR.predict(X_crosstest_svd)202print("LinearRegression_RMSE:",np.sqrt(mean_squared_error(y_crosstest,pred_LR)))203204print(datetime.now() - start)205206#decision tree along with Adaboost207start = datetime.now() 208from sklearn.tree import DecisionTreeRegressor209from sklearn.ensemble import AdaBoostRegressor210AR = AdaBoostRegressor(DecisionTreeRegressor(max_depth = 100),n_estimators = 100, random_state=120).fit(X_crosstrain_scl,y_crosstrain)211pred_AR = AR.predict(X_crosstest_scl)212print("AdaboostDecisionTreeRegression_RMSE:",np.sqrt(mean_squared_error(y_crosstest,pred_AR)))213print(datetime.now() - start)214215216#***********************************Regular Features**************************************************217218def findword(str1, str2):219 return sum(int(str2.find(word)>=0) for word in str1.split())220 221#Features in Train set222Train_all['length_pdt'] = Train_all['product_title'].apply(len)223Train_all['length_st'] = Train_all['search_term'].apply(len) 224Train_all['length_desc'] = Train_all['product_description'].apply(len)225Train_all['Length_Brand'] = Train_all['Brand'].apply(len)226Train_all['search_in_title'] = Train_all['prodtext1'].map(lambda x:findword(x.split('\t')[0],x.split('\t')[1]))227Train_all['search_in_description'] = Train_all['prodtext1'].map(lambda x:findword(x.split('\t')[0],x.split('\t')[2]))228Train_all['search_in_brand'] = Train_all['prodtext1'].map(lambda x:findword(x.split('\t')[0],x.split('\t')[3]))229Train_all['Ratio_title'] = Train_all['search_in_title']/Train_all['length_st']230Train_all['Ratio_desc'] = Train_all['search_in_description']/Train_all['length_st']231Train_all['Ratio_brand'] = Train_all['search_in_brand']/Train_all['length_st']232233Train_all.head()234 # Exploratory Data Analysis235Histgram_pdt = Train_all['length_pdt'].plot(bins=50,kind='hist') # Normal236Histgram_st = Train_all['length_st'].plot(bins=50,kind='hist',color='green') #Normal237Histgram_desc = Train_all['length_desc'].plot(bins=100,kind='hist',color='purple') # Right Skwed238Histgram_searchttitle = Train_all['search_in_title'].plot(kind='hist',color='blue')239Histgram_searchbrand = Train_all['search_in_brand'].plot(kind='hist',color='black')240Histgram_Ratiotitle = Train_all['Ratio_title'].plot(kind='hist',color='purple')241242# Summary statistics for engineered column - length243print(Train_all['length_pdt'].describe())244print(Train_all['length_st'].describe())245print(Train_all['length_desc'].describe())246247# Check the lenghtiest product title and search term individually248print(Train_all[Train_all['length_pdt'] == 147]['product_title'])249print(Train_all[Train_all['length_st'] == 60]['search_term'])250251252# Histogram of relevance vs lenght of product title and search term253print(Train_all.hist(column='length_pdt',by ='relevance',bins = 50, figsize=(15,6)))254print(Train_all.hist(column='length_st',by ='relevance',bins = 50, figsize=(15,6)))255print(Train_all.hist(column='length_desc',by ='relevance',bins = 100, figsize=(15,6)))256print(Train_all.hist(column='search_in_title',by ='relevance',bins = 10, figsize=(15,6)))257print(Train_all.hist(column='search_in_brand',by ='relevance',bins = 10, figsize=(15,6)))258print(Train_all.hist(column='Ratio_title',by ='relevance',bins = 10, figsize=(15,6)))259260#Features in Test set261Test_all['length_pdt'] = Test_all['product_title'].apply(len)262Test_all['length_st'] = Test_all['search_term'].apply(len)263Test_all['length_desc'] = Test_all['product_description'].apply(len)264Test_all['Length_Brand'] = Test_all['Brand'].apply(len)265Test_all['search_in_title'] = Test_all['prodtext1'].map(lambda x:findword(x.split('\t')[0],x.split('\t')[1]))266Test_all['search_in_description'] = Test_all['prodtext1'].map(lambda x:findword(x.split('\t')[0],x.split('\t')[2]))267Test_all['search_in_brand'] = Test_all['prodtext1'].map(lambda x:findword(x.split('\t')[0],x.split('\t')[3]))268Test_all['Ratio_title'] = Test_all['search_in_title']/Test_all['length_st']269Test_all['Ratio_desc'] = Test_all['search_in_description']/Test_all['length_st']270Test_all['Ratio_brand'] = Test_all['search_in_brand']/Test_all['length_st']271272X1_train = Train_all.drop(['id','product_uid','relevance','product_title','prodtext','prodtext1','search_term','product_description','Brand'],axis=1)273X1_test = Test_all.drop(['id','product_uid','product_title','prodtext','search_term','prodtext1','product_description','Brand'],axis=1)274Y1_train = Train_all['relevance']275276#Cross validation277start = datetime.now() 278279X1,y1 = shuffle(X1_train,Y1_train,random_state = 13)280offset = int(X1.shape[0] *0.80)281X1_crosstrain , y1_crosstrain = X1[:offset],y1[:offset]282X1_crosstest , y1_crosstest = X1[offset:],y1[offset:]283284#Calculate RMSE for Random Forest285RF1cross = ensemble.RandomForestRegressor(n_estimators = 500,verbose=1,n_jobs=-1,random_state = 120,max_depth=16)286RF1cross_fit = RF1cross.fit(X1_crosstrain,y1_crosstrain)287RF1mse = RF1cross_fit.predict(X1_crosstest)288print("Random Forest:",np.sqrt(mean_squared_error(y1_crosstest,RF1mse)))289print(datetime.now() - start)290291start = datetime.now() 292from sklearn.svm import SVR293from sklearn.metrics import mean_squared_error294Model1_one = SVR(C=1.0, kernel='rbf', degree=3, gamma='auto', coef0=0.0, shrinking=True, 295 tol=0.001, cache_size=200,verbose=False, max_iter=30000)296Model1_one.fit(X1_crosstrain,y1_crosstrain)297SVM1mse = Model1_one.predict(X1_crosstest)298print("SVM_RMSE:",np.sqrt(mean_squared_error(y1_crosstest,SVM1mse)))299print(datetime.now() - start)300301start = datetime.now() 302from sklearn.neighbors import KNeighborsRegressor303Model2_two = KNeighborsRegressor(n_neighbors=31)304Model2_two.fit(X1_crosstrain,y1_crosstrain)305KNN1mse = Model2_two.predict(X1_crosstest)306print("KNN_RMSE:",np.sqrt(mean_squared_error(y1_crosstest,KNN1mse)))307print(datetime.now() - start)308 309## PCA Analysis for Regular Features310from sklearn.preprocessing import scale311from sklearn.decomposition import PCA312pca = PCA()313X1_reduced = pca.fit_transform(scale(X1_crosstrain))314315np.cumsum(np.round(pca.explained_variance_ratio_,decimals=4)*100) # 12 components explain 90% of the variation316plt.clf()317plt.plot(pca.explained_variance_,linewidth=2)318plt.xlabel('n_components')319plt.ylabel('explained_variance')320321RandomForest = ensemble.RandomForestRegressor(verbose=1,n_jobs=-1,random_state = 120)322323start = datetime.now()324from sklearn.pipeline import Pipeline325from sklearn.grid_search import GridSearchCV326327pipe = Pipeline(steps=[('pca', pca), ('RandomForest', RandomForest)])328estimator = GridSearchCV(pipe,dict(pca__n_components=[9,12,15],329 RandomForest__n_estimators=[250,500,750]))330331estimator.fit(X1_crosstrain,y1_crosstrain)332RF2mse = estimator.predict(X1_crosstest)333print("RF2_RMSE:",np.sqrt(mean_squared_error(y1_crosstest,RF2mse)))334print(datetime.now() - start)335336Train = tfidf.fit_transform(X_train)337338Test = tfidf.transform(X_test)339340from sklearn.decomposition import TruncatedSVD341from sklearn.preprocessing import StandardScaler342343svd = TruncatedSVD(n_components=300, algorithm='randomized', n_iter=5, random_state=None, tol=0.0)344X_crosstrain_svd = svd.fit_transform(Train)345svd.explained_variance_ratio_.sum()346X_crosstest_svd = svd.transform(Test)347348scl = StandardScaler(copy=True, with_mean=True, with_std=True)349350X_crosstrain_scl = scl.fit_transform(X_crosstrain_svd)351X_crosstest_scl = scl.transform(X_crosstest_svd)352353start = datetime.now()354start = datetime.now() 355from sklearn import ensemble356Model_three = ensemble.RandomForestRegressor(n_estimators = 500,verbose=1,n_jobs=-1,random_state = 120,max_depth=16)357Model_three.fit(X_crosstrain_scl,Y_train)358RFmse = Model_three.predict(X_crosstest_scl)359#print("RandomForest_RMSE:",np.sqrt(mean_squared_error(y_crosstest,RFmse)))360print(datetime.now() - start)361362start = datetime.now()363RF1cross = ensemble.RandomForestRegressor(n_estimators = 500,verbose=1,n_jobs=-1,random_state = 120,max_depth=16)364RF1cross_fit = RF1cross.fit(X1_train,Y1_train)365RF1mse = RF1cross_fit.predict(X1_test)366#print("Random Forest:",np.sqrt(mean_squared_error(y1_crosstest,RF1mse)))367WA = (RF1mse+RFmse)/2368pd.DataFrame({"id": Test_all.id, "relevance": WA}).to_csv('Relevance_file.csv',index=False)369print(datetime.now() - start)
...
test_sorters.py
Source:test_sorters.py
1from spikeforestsorters import MountainSort4, SpykingCircus, KiloSort, KiloSort2, IronClust, HerdingSpikes2, JRClust, Tridesclous, Klusta, Waveclus, YASS, YASS12from mountaintools import client as mt3import spikeforest_analysis as sa4import numpy as np5import pytest6synth_magland_c4_recdir = 'sha1dir://fb52d510d2543634e247e0d2d1d4390be9ed9e20.synth_magland/datasets_noise10_K10_C4/001_synth'7synth_magland_c8_recdir = 'sha1dir://fb52d510d2543634e247e0d2d1d4390be9ed9e20.synth_magland/datasets_noise10_K10_C8/001_synth'8kampff1_recdir = 'sha1dir://c86202ca09f303b6c6d761b94975054c29c85d2b.paired_kampff/kampff1'9neuropix32c_recdir = 'sha1dir://d446c8e74fc4ca3a0dab491fca6c10189b527709.neuropix32c.c14'10boyden32c_recdir = 'sha1dir://b28dbf52748dcb401034d1c353807bcbff20e106.boyden32c.1103_1_1'11sqmea64c_recdir = 'sha1dir://e8de6ac2138bf775f29f8ab214d04aa92e20ca79'12paired_mea64c_recdir = 'sha1dir://7f12606802ade3c7c71eb306490b7840eb8b1fb4.paired_mea64c'13neurocube1c_recdir = 'sha1dir://e6cb8f3bb5228c73208a82d2854552af38ab6b40'14visapy30c_recdir = 'sha1dir://97253adc2581b1acbf9a9fffcbc00247d8088a1d.mea_c30.set1'15# synth_bionet_static1_recdir = 'sha1dir://abc900f5cd62436e7c89d914c9f36dcd7fcca0e7.synth_bionet/bionet_static/static_8x_C_4B'16# synth_bionet_static1_recdir = '/mnt/home/jjun/ceph/recordings/bionet_static_rec1'17# synth_bionet_static1_recdir = '/mnt/home/jjun/ceph/groundtruth/bionet/bionet_static/static_8x_A_4A'18synth_bionet_static1_recdir = '/mnt/home/jjun/ceph/groundtruth/bionet/bionet_static/static_8x_C_4B'19hybrid_janelia_static1_recdir = '/mnt/home/jjun/ceph/groundtruth/hybrid_synth/static_siprobe/rec_64c_1200s_11'20def main():21 mem_profile_test(IronClust, dict(fSave_spkwav=True), synth_magland_c8_recdir)22 mem_profile_test(IronClust, dict(fSave_spkwav=False), synth_magland_c8_recdir)23@pytest.mark.spikeforest24@pytest.mark.irc_hybrid_static25@pytest.mark.test_all26@pytest.mark.exclude27def test_irc_hybrid_static():28 sorter = IronClust29 params = dict()30 do_sorting_test(sorter, params, hybrid_janelia_static1_recdir,31 assert_avg_accuracy=0.5, _keep_temp_files=True)32@pytest.mark.spikeforest33@pytest.mark.irc_bionet_static134@pytest.mark.test_all35@pytest.mark.exclude36def test_irc_bionet_static1():37 sorter = IronClust38 params = dict()39 do_sorting_test(sorter, params, synth_bionet_static1_recdir,40 assert_avg_accuracy=0.5, _keep_temp_files=True)41@pytest.mark.spikeforest42@pytest.mark.yass1_visapy30c43@pytest.mark.test_all44@pytest.mark.exclude45def test_yass1_visapy30c():46 sorter = YASS147 params = dict(48 detect_sign=-1,49 )50 do_sorting_test(sorter, params, visapy30c_recdir, assert_avg_accuracy=0.1, _keep_temp_files=True, container='default')51@pytest.mark.spikeforest52@pytest.mark.yass_visapy30c53@pytest.mark.test_all54@pytest.mark.exclude55def test_yass_visapy30c():56 sorter = YASS57 params = dict(58 detect_sign=-1,59 )60 do_sorting_test(sorter, params, visapy30c_recdir, assert_avg_accuracy=0.1, _keep_temp_files=True)61@pytest.mark.spikeforest62@pytest.mark.waveclus_neurocube1c63@pytest.mark.test_all64@pytest.mark.exclude65def test_waveclus_neurocube1c():66 sorter = Waveclus67 params = dict()68 do_sorting_test(sorter, params, neurocube1c_recdir, assert_avg_accuracy=0.1, _keep_temp_files=True)69@pytest.mark.spikeforest70@pytest.mark.ms4_neurocube1c71@pytest.mark.test_all72@pytest.mark.exclude73def test_ms4_neurocube1c():74 sorter = MountainSort475 sorter = MountainSort476 params = dict(77 detect_sign=-1,78 adjacency_radius=5079 )80 do_sorting_test(sorter, params, neurocube1c_recdir, assert_avg_accuracy=0.2)81@pytest.mark.spikeforest82@pytest.mark.irc_neurocube1c83@pytest.mark.test_all84@pytest.mark.exclude85def test_irc_neurocube1c():86 sorter = IronClust87 params = dict()88 do_sorting_test(sorter, params, neurocube1c_recdir, assert_avg_accuracy=0.1, _keep_temp_files=True)89@pytest.mark.spikeforest90@pytest.mark.ms491@pytest.mark.test_all92@pytest.mark.exclude93def test_ms4():94 sorter = MountainSort495 params = dict(96 detect_sign=-1,97 adjacency_radius=5098 )99 # do_sorting_test(sorter, params, synth_magland_c4_recdir, assert_avg_accuracy=0.8)100 do_sorting_test(sorter, params, synth_magland_c8_recdir, assert_avg_accuracy=0.8)101 # do_sorting_test(sorter, params, kampff1_recdir, assert_avg_accuracy=0.8) # jfm laptop: ~220 seconds102@pytest.mark.spikeforest103@pytest.mark.ms4_magland_c4104@pytest.mark.test_all105@pytest.mark.exclude106def test_ms4_magland_c4():107 sorter = MountainSort4108 params = dict(109 detect_sign=-1,110 adjacency_radius=75,111 )112 do_sorting_test(sorter, params, synth_magland_c4_recdir, assert_avg_accuracy=0.5)113@pytest.mark.spikeforest114@pytest.mark.ms4_magland_c8115@pytest.mark.test_all116@pytest.mark.exclude117def test_ms4_magland_c8():118 sorter = MountainSort4119 params = dict(120 detect_sign=-1,121 adjacency_radius=75,122 )123 do_sorting_test(sorter, params, synth_magland_c8_recdir, assert_avg_accuracy=0.5)124@pytest.mark.spikeforest125@pytest.mark.ms4_neuropix32c126@pytest.mark.test_all127@pytest.mark.exclude128def test_ms4_neuropix32c():129 sorter = MountainSort4130 params = dict(131 detect_sign=-1,132 adjacency_radius=75,133 )134 do_sorting_test(sorter, params, neuropix32c_recdir, assert_avg_accuracy=0.5)135@pytest.mark.spikeforest136@pytest.mark.sc_magland_c8137@pytest.mark.test_all138@pytest.mark.exclude139def test_sc():140 sorter = SpykingCircus141 params = dict(142 detect_sign=-1,143 )144 do_sorting_test(sorter, params, synth_magland_c8_recdir, assert_avg_accuracy=0.4)145@pytest.mark.spikeforest146@pytest.mark.ks2_hybrid_static147@pytest.mark.test_all148@pytest.mark.exclude149def test_ks2_hybrid_static():150 sorter = KiloSort2151 params = dict()152 do_sorting_test(sorter, params, hybrid_janelia_static1_recdir, assert_avg_accuracy=0.8, _keep_temp_files=True)153@pytest.mark.spikeforest154@pytest.mark.ks_hybrid_static155@pytest.mark.test_all156@pytest.mark.exclude157def test_ks_hybrid_static():158 sorter = KiloSort159 params = dict()160 do_sorting_test(sorter, params, hybrid_janelia_static1_recdir, assert_avg_accuracy=0.8, _keep_temp_files=True)161@pytest.mark.spikeforest162@pytest.mark.ks2_magland_c4163@pytest.mark.test_all164@pytest.mark.exclude165def test_ks2_magland_c4():166 sorter = KiloSort2167 params = dict()168 do_sorting_test(sorter, params, synth_magland_c4_recdir, assert_avg_accuracy=0.8, _keep_temp_files=True)169@pytest.mark.spikeforest170@pytest.mark.ks2_magland_c8171@pytest.mark.test_all172@pytest.mark.exclude173def test_ks2_magland_c8():174 sorter = KiloSort2175 params = dict()176 do_sorting_test(sorter, params, synth_magland_c8_recdir, assert_avg_accuracy=0.5, _keep_temp_files=True)177@pytest.mark.spikeforest178@pytest.mark.ks_magland_c8179@pytest.mark.test_all180@pytest.mark.exclude181def test_ks_magland_c8():182 sorter = KiloSort183 params = dict()184 do_sorting_test(sorter, params, synth_magland_c8_recdir, assert_avg_accuracy=0.5, _keep_temp_files=True)185@pytest.mark.spikeforest186@pytest.mark.ks_boyden32c187@pytest.mark.test_all188@pytest.mark.exclude189def test_ks_boyden32c():190 sorter = KiloSort191 params = dict()192 do_sorting_test(sorter, params, boyden32c_recdir, assert_avg_accuracy=0.5, _keep_temp_files=True)193@pytest.mark.spikeforest194@pytest.mark.klusta_magland_c4195@pytest.mark.test_all196@pytest.mark.exclude197def test_klusta_magland_c4():198 sorter = Klusta199 params = dict(200 detect_sign=-1,201 adjacency_radius=50202 )203 do_sorting_test(sorter, params, synth_magland_c4_recdir, assert_avg_accuracy=0.8)204@pytest.mark.spikeforest205@pytest.mark.klusta_magland_c8206@pytest.mark.test_all207@pytest.mark.exclude208def test_klusta_magland_c8():209 sorter = Klusta210 params = dict(211 detect_sign=-1,212 adjacency_radius=50213 )214 do_sorting_test(sorter, params, synth_magland_c8_recdir, assert_avg_accuracy=0.8)215@pytest.mark.spikeforest216@pytest.mark.tridesclous_magland_c4217@pytest.mark.test_all218@pytest.mark.exclude219def test_tridesclous_magland_c4():220 sorter = Tridesclous221 params = dict()222 do_sorting_test(sorter, params, synth_magland_c4_recdir, assert_avg_accuracy=0.5, container='default')223@pytest.mark.spikeforest224@pytest.mark.tridesclous_magland_c8225@pytest.mark.test_all226@pytest.mark.exclude227def test_tridesclous_magland_c8():228 sorter = Tridesclous229 params = dict()230 do_sorting_test(sorter, params, synth_magland_c8_recdir, assert_avg_accuracy=0.1, container='default')231@pytest.mark.spikeforest232@pytest.mark.hs2_magland_c8233@pytest.mark.test_all234@pytest.mark.exclude235def test_hs2_magland_c8():236 sorter = HerdingSpikes2237 params = dict()238 do_sorting_test(sorter, params, synth_magland_c8_recdir, assert_avg_accuracy=0.1, container='default')239@pytest.mark.spikeforest240@pytest.mark.hs2_paired_mea64c241@pytest.mark.test_all242@pytest.mark.exclude243def test_hs2_paired_mea64c():244 sorter = HerdingSpikes2245 params = dict()246 do_sorting_test(sorter, params, paired_mea64c_recdir, assert_avg_accuracy=0.1)247@pytest.mark.spikeforest248@pytest.mark.hs2_neuropix32c249@pytest.mark.test_all250@pytest.mark.exclude251def test_hs2_neuropix32c():252 sorter = HerdingSpikes2253 params = dict(254 adjacency_radius=50,255 )256 do_sorting_test(sorter, params, synth_magland_c8_recdir, assert_avg_accuracy=0.1)257@pytest.mark.spikeforest258@pytest.mark.hs2_visapy30c259@pytest.mark.test_all260@pytest.mark.exclude261def test_hs2_visapy30c():262 sorter = HerdingSpikes2263 params = dict()264 do_sorting_test(sorter, params, visapy30c_recdir, assert_avg_accuracy=0.1, container='default')265@pytest.mark.spikeforest266@pytest.mark.hs2_boyden32c267@pytest.mark.test_all268@pytest.mark.exclude269def test_hs2_boyden32c():270 sorter = HerdingSpikes2271 params = dict(272 adjacency_radius=50,273 )274 do_sorting_test(sorter, params, boyden32c_recdir, assert_avg_accuracy=0.1)275@pytest.mark.spikeforest276@pytest.mark.hs2_sqmea64c277@pytest.mark.test_all278@pytest.mark.exclude279def test_hs2_sqmea64c():280 sorter = HerdingSpikes2281 params = dict(282 adjacency_radius=50,283 )284 do_sorting_test(sorter, params, sqmea64c_recdir, assert_avg_accuracy=0.1)285@pytest.mark.spikeforest286@pytest.mark.ks2_neuropix32c287@pytest.mark.test_all288@pytest.mark.exclude289def test_ks2_neuropix32c():290 sorter = KiloSort2291 params = dict()292 do_sorting_test(sorter, params, neuropix32c_recdir, assert_avg_accuracy=0.2)293@pytest.mark.spikeforest294@pytest.mark.ks2_boyden32c295@pytest.mark.test_all296@pytest.mark.exclude297def test_ks2_boyden32c():298 sorter = KiloSort2299 params = dict()300 do_sorting_test(sorter, params, boyden32c_recdir, assert_avg_accuracy=0.2)301@pytest.mark.spikeforest302@pytest.mark.ks2_sqmea64c303@pytest.mark.test_all304@pytest.mark.exclude305def test_ks2_sqmea64c():306 sorter = KiloSort2307 params = dict()308 do_sorting_test(sorter, params, sqmea64c_recdir, assert_avg_accuracy=0.2)309@pytest.mark.spikeforest310@pytest.mark.irc_neuropix32c311@pytest.mark.test_all312@pytest.mark.exclude313def test_irc_neuropix32c():314 sorter = IronClust315 params = dict()316 do_sorting_test(sorter, params, neuropix32c_recdir, assert_avg_accuracy=0.2)317@pytest.mark.spikeforest318@pytest.mark.irc_magland_c8319@pytest.mark.test_all320@pytest.mark.exclude321def test_irc_magland_c8():322 sorter = IronClust323 params = dict()324 do_sorting_test(sorter, params, synth_magland_c8_recdir, assert_avg_accuracy=0.2)325@pytest.mark.spikeforest326@pytest.mark.irc_magland_c4327@pytest.mark.test_all328@pytest.mark.exclude329def test_irc_magland_c4():330 sorter = IronClust331 params = dict()332 do_sorting_test(sorter, params, synth_magland_c4_recdir, assert_avg_accuracy=0.2)333@pytest.mark.spikeforest334@pytest.mark.irc_sqmea64c335@pytest.mark.test_all336@pytest.mark.exclude337def test_irc_sqmea64c():338 sorter = IronClust339 params = dict()340 do_sorting_test(sorter, params, sqmea64c_recdir, assert_avg_accuracy=0.1)341@pytest.mark.spikeforest342@pytest.mark.jrc_magland_c8343@pytest.mark.test_all344@pytest.mark.exclude345def test_jrc_magland_c8():346 sorter = JRClust347 params = dict()348 do_sorting_test(sorter, params, synth_magland_c8_recdir, assert_avg_accuracy=0.1)349@pytest.mark.spikeforest350@pytest.mark.jrc_neuropix32c351@pytest.mark.test_all352@pytest.mark.exclude353def test_jrc_neuropix32c():354 sorter = JRClust355 params = dict(356 detect_sign=-1,357 adjacency_radius=75,358 )359 do_sorting_test(sorter, params, neuropix32c_recdir, assert_avg_accuracy=0.5)360@pytest.mark.spikeforest361@pytest.mark.jrc_visapy30c362@pytest.mark.test_all363@pytest.mark.exclude364def test_jrc_visapy30c():365 sorter = JRClust366 params = dict()367 do_sorting_test(sorter, params, visapy30c_recdir, assert_avg_accuracy=0.1, _keep_temp_files=True, container='default')368@pytest.mark.spikeforest369@pytest.mark.ks2_kampff370@pytest.mark.test_all371@pytest.mark.exclude372def test_ks2_kampff():373 sorter = KiloSort2374 params = dict()375 do_sorting_test(sorter, params, kampff1_recdir, assert_avg_accuracy=0.8)376def mem_profile_test(sorting_processor, params, recording_dir, container='default', _keep_temp_files=True):377 mt.configDownloadFrom('spikeforest.public')378 params['fMemProfile'] = True379 recdir = recording_dir380 mt.createSnapshot(path=recdir, download_recursive=True)381 sorting = sorting_processor.execute(382 recording_dir=recdir,383 firings_out={'ext': '.mda'},384 **params,385 _container=container,386 _force_run=True,387 _keep_temp_files=_keep_temp_files388 )389def do_sorting_test(sorting_processor, params, recording_dir, assert_avg_accuracy, container='default', _keep_temp_files=False):390 mt.configDownloadFrom('spikeforest.public')391 recdir = recording_dir392 mt.createSnapshot(path=recdir, download_recursive=True)393 sorting = sorting_processor.execute(394 recording_dir=recdir,395 firings_out={'ext': '.mda'},396 **params,397 _container=container,398 _force_run=True,399 _keep_temp_files=_keep_temp_files400 )401 comparison = sa.GenSortingComparisonTable.execute(402 firings=sorting.outputs['firings_out'],403 firings_true=recdir + '/firings_true.mda',404 units_true=[],405 json_out={'ext': '.json'},406 html_out={'ext': '.html'},407 _container='default',408 _force_run=True409 )410 X = mt.loadObject(path=comparison.outputs['json_out'])411 accuracies = [float(a['accuracy']) for a in X.values()]412 avg_accuracy = np.mean(accuracies)413 print('Average accuracy: {}'.format(avg_accuracy))414 assert(avg_accuracy >= assert_avg_accuracy)415if __name__ == "__main__":...
Test.py
Source:Test.py
1'''2Place to test individual transforms.3Note: the newest tests tend to be near the top.4'''5from pathlib import Path6# Import all transform functions.7import Framework8from Plugins import *9this_dir = Path(__file__).resolve().parents[1]10Settings(11 extension_name = 'test_customizer',12 path_to_x4_folder = r'D:\Games\Steam\SteamApps\common\X4 Foundations',13 developer = 1,14 )15# For all tests to run, mostly to tease out exceptions after16# code changes. Note: some tests may be a bit outdated.17test_all = 018if 0:19 GUI.Start_GUI()20# Test sector resizing.21if 0 or test_all:22 Scale_Sector_Size(23 scaling_factor = 0.4, 24 scaling_factor_2 = 0.3, 25 transition_size_start = 200000,26 transition_size_end = 400000,27 recenter_sectors = False,28 precision_steps = 10,29 remove_ring_highways = True,30 remove_nonring_highways = False,31 extra_scaling_for_removed_highways = 0.7,32 _test = False33 )34 35# Test exe edits.36if 0:37 Remove_Sig_Errors()38if 0:39 Remove_Modified()40if 0:41 High_Precision_Systemtime()42 43if 0 or test_all:44 Increase_AI_Script_Waits(45 oos_multiplier = 2,46 oos_seta_multiplier = 4,47 oos_max_wait = 15,48 iv_multiplier = 1,49 iv_seta_multiplier = 2,50 iv_max_wait = 5,51 filter = '*',52 include_extensions = False,53 skip_combat_scripts = False,54 )55if 0 or test_all:56 Adjust_OOS_Damage(0.5)57# Diff generator test.58if 0 or test_all:59 Generate_Diff(60 original_file_path = this_dir / '../private/test' / 'test_original_node.xml',61 modified_file_path = this_dir / '../private/test' / 'test_modified_node.xml',62 output_file_path = this_dir / '../private/test' / 'test_patch_node.xml',63 )64 Generate_Diffs(65 original_dir_path = this_dir / '../private/test' / 'orig_files',66 modified_dir_path = this_dir / '../private/test' / 'mod_files',67 output_dir_path = this_dir / '../private/test' / 'diff_files',68 )69 70if 0 or test_all:71 # Try forcing an attribute.72 Settings(forced_xpath_attributes = 'id,method,tags')73 Generate_Diffs(74 original_dir_path = this_dir / '../private/test/deadair/orig',75 modified_dir_path = this_dir / '../private/test/deadair/mod',76 output_dir_path = this_dir / '../private/test/deadair/diff',77 )78 79# TODO: delete this, or fix god_edit error (bad xml, <object> closed by </stations>).80#if 0 or test_all:81# Generate_Diff(82# original_file_path = this_dir / '../private/test/god_orig.xml',83# modified_file_path = this_dir / '../private/test/god_edit.xml',84# output_file_path = this_dir / '../private/test/god_diff.xml',85# )86# Test the extension checker.87#if 0 or test_all:88# Check_Extension('test_mod')89if 0 or test_all:90 # Alternatively, check everything (may take longer).91 Check_All_Extensions()92if 0 or test_all:93 Color_Text((20005,3012,'C'))94if 0 or test_all:95 # Live editor tree builders.96 edit_tree = Framework.Live_Editor.Get_Tree_View('components')97 edit_tree = Framework.Live_Editor.Get_Tree_View('weapons')98if 0 or test_all:99 edit_tree = Framework.Live_Editor.Get_Tree_View('ships')100if 0 or test_all:101 Adjust_Mission_Reward_Mod_Chance(10)102# Ship transforms.103if 0 or test_all:104 Adjust_Ship_Speed(105 ('name ship_xen_xl_carrier_01_a*', 1.2),106 ('class ship_s' , 2.0),107 ('type corvette' , 1.5),108 ('purpose fight' , 1.2),109 ('*' , 1.1) )110 Adjust_Ship_Crew_Capacity(111 ('class ship_xl' , 2.0),112 ('*' , 1.5)113 )114 Adjust_Ship_Drone_Storage(115 ('class ship_xl' , 2.0),116 ('*' , 1.5)117 )118 Adjust_Ship_Missile_Storage(119 ('class ship_xl' , 2.0),120 ('*' , 1.5)121 )122 Adjust_Ship_Hull(123 ('class ship_xl' , 2.0),124 ('*' , 1.5)125 )126 Adjust_Ship_Turning(127 ('class ship_xl' , 2.0),128 ('*' , 1.5)129 )130 131 Print_Ship_Stats('ship_stats_postmod')132if 0:133 Adjust_Ship_Hull(134 ('class ship_l' , 1.5), 135 ('class ship_xl', 1.5))136 137if 0 or test_all:138 Set_Default_Radar_Ranges(139 ship_xl = 50,140 ship_l = 40,141 ship_m = 30,142 ship_s = 20,143 ship_xs = 20,144 spacesuit = 20,145 station = 40,146 satellite = 30,147 adv_satellite = 50,148 )149 Set_Ship_Radar_Ranges(150 ('type scout', 40),151 )152if 0 or test_all:153 Disable_AI_Travel_Drive()154 Remove_Engine_Travel_Bonus()155if 0 or test_all:156 Rebalance_Engines(purpose_speed_mults = None, adjust_cargo = True)157 Rebalance_Engines(race_speed_mults = None, adjust_cargo = True)158if 0 or test_all:159 Adjust_Engine_Boost_Duration(0.2)160 Adjust_Engine_Boost_Speed(0.25)161if 0 or test_all:162 # Adjust speeds per ship class.163 # Note: vanilla averages and ranges are: 164 # xs: 130 (58 to 152)165 # s : 328 (71 to 612)166 # m : 319 (75 to 998)167 # l : 146 (46 to 417)168 # xl: 102 (55 to 164)169 # Try clamping variation to within 0.5x (mostly affects medium).170 Rescale_Ship_Speeds(171 # Ignore the python (unfinished).172 {'match_any' : ['name ship_spl_xl_battleship_01_a_macro'], 'skip' : True, 'use_arg_engine' : True},173 {'match_all' : ['type scout' ], 'average' : 500, 'variation' : 0.2, 'use_arg_engine' : True},174 {'match_all' : ['class ship_s'], 'average' : 400, 'variation' : 0.5, 'use_arg_engine' : True},175 {'match_all' : ['class ship_m'], 'average' : 300, 'variation' : 0.5, 'use_arg_engine' : True},176 {'match_all' : ['class ship_l'], 'average' : 200, 'variation' : 0.5, 'use_arg_engine' : True},177 {'match_all' : ['class ship_xl'], 'average' : 150, 'variation' : 0.5, 'use_arg_engine' : True})178if 0 or test_all:179 Adjust_Ship_Cargo_Capacity(180 {'match_all' : ['purpose mine'], 'multiplier' : 2,},181 {'match_all' : ['purpose trade'], 'multiplier' : 1.5},182 )183 184# Test the gui live editor, doing a transform before and after185# the patch application. Transform before should show up in the186# gui edit tables; transform after should show up in the final187# game files (along with the hand edits from the gui).188if 0:189 # Pre-editor should have halved damage, post-editor 2x damage,190 # compared to vanilla or the input extensions.191 #Adjust_Weapon_Damage(0.5)192 Apply_Live_Editor_Patches()193 #Adjust_Weapon_Damage(4)194if 0 or test_all:195 Adjust_Mission_Rewards(0.5)196 #Write_To_Extension()197# Ware transforms and printout.198if 0 or test_all:199 #Print_Ware_Stats('ware_stats_premod')200 Adjust_Ware_Price_Spread(201 ('id energycells' , 2 ),202 ('group shiptech' , 0.8),203 ('container ship' , 1.5),204 ('tags crafting' , 0.2),205 ('*' , 0.1) )206 Adjust_Ware_Prices(207 ('container inventory' , 0.5) )208 #Print_Ware_Stats('ware_stats_postmod')209# Old style weapon edits.210if 0 or test_all:211 Adjust_Weapon_Damage(1.5)212 Adjust_Weapon_Damage(213 ('tags small standard weapon' , 2),214 ('*' , 1.2),215 )216 Adjust_Weapon_Range(217 ('tags small standard weapon' , 2),218 ('tags missile' , 2),219 )220 Adjust_Weapon_Shot_Speed(221 ('tags small standard weapon' , 2),222 ('tags missile' , 2),223 )224 Adjust_Weapon_Fire_Rate(225 ('tags small standard weapon' , 2),226 ('tags missile' , 2),227 )228# Weapon transforms and printout.229if 0 or test_all:230 #Print_Weapon_Stats('weapon_stats_premod')231 Adjust_Weapon_Damage(1.5)232 Adjust_Weapon_Damage(233 {'match_all' : ['tags small standard weapon'], 'multiplier' : 2},234 {'match_all' : ['*'], 'multiplier' : 1.2},235 )236 Adjust_Weapon_Range(237 {'match_all' : ['tags small standard weapon'], 'multiplier' : 2},238 {'match_all' : ['tags missile'], 'multiplier' : 2},239 )240 Adjust_Weapon_Shot_Speed(241 {'match_all' : ['tags small standard weapon'], 'multiplier' : 2},242 {'match_all' : ['tags missile'], 'multiplier' : 2},243 )244 Adjust_Weapon_Fire_Rate(245 {'match_all' : ['tags small standard weapon'], 'multiplier' : 2},246 {'match_all' : ['tags missile'], 'multiplier' : 2},247 )248 Adjust_Weapon_Fire_Rate(249 {'match_all' : ['tags small standard weapon'], 'multiplier' : 2, 'min' : 0.5},250 {'match_all' : ['tags missile'], 'multiplier' : 1.5},251 )252 #Print_Weapon_Stats('weapon_stats_postmod')253 254# Testing ways to call Jobs.255if 0 or test_all: 256 Adjust_Job_Count(257 ('id masstraffic*' , 0.5),258 ('tags military destroyer', 2 ),259 ('tags miner' , 1.5),260 ('size s' , 1.5),261 ('faction argon' , 1.2),262 ('*' , 1.1) )263 264# Simple cat unpack, allowing errors.265if 0 or test_all:266 Settings.allow_cat_md5_errors = True267 Cat_Unpack(268 source_cat_path = r'D:\X4\Pack',269 dest_dir_path = r'D:\X4\UnPack',270 )271# Slightly more complex cat unpack.272if 0 or test_all:273 # Pick where to grab cats from.274 # Could also call this script from that directory and use relative275 # paths for the cat file names.276 # Append the name of a cat file if wanting to unpack just one.277 cat_dir = Path(r'C:\Steam\SteamApps\common\X4 Foundations')278 # Pick the output folder. This script places it under the cat_dir.279 dest_dir_path = 'extracted'280 # Optional wildcard pattern to use for matching.281 # Just lua for quick test.282 include_pattern = ['*.lua']#['*.xml','*.xsd'] #,'*.xpl']283 exclude_pattern = None284 # Call the unpacker.285 Cat_Unpack(286 source_cat_path = cat_dir,287 #dest_dir_path = cat_dir / dest_dir_path,288 dest_dir_path = r'D:\X4_extracted',289 include_pattern = include_pattern,290 exclude_pattern = exclude_pattern291 )292# Cat pack test.293if 0 or test_all:294 # Pick where to grab files from.295 # Could also call this script from that directory and use relative296 # paths for the cat file names.297 dir_path = Path(r'C:\Steam\SteamApps\common\X4 Foundations\extensions\test_mod')298 # Optional wildcard pattern to use for matching.299 include_pattern = '*.xml'300 exclude_pattern = None301 # Name of the cat file.302 # For extensions, use prefix 'ext_' for patching game files, or303 # prefix 'subst_' for overwriting game files.304 cat_name = 'ext_01.cat'305 Cat_Pack(306 source_dir_path = dir_path,307 dest_cat_path = dir_path / cat_name,308 include_pattern = include_pattern,309 exclude_pattern = exclude_pattern,310 generate_sigs = True,311 separate_sigs = True,312 )313 314# Run diff patch test on whatever xml.315if 0 or test_all:316 jobs_game_file = Framework.Load_File('libraries/jobs.xml')317 Framework.File_Manager.XML_Diff.Unit_Test(318 test_node = jobs_game_file.Get_Root(), 319 # Shorten test count when in test_all mode.320 num_tests = 100 if not test_all else 5, 321 edits_per_test = 5,322 rand_seed = 1,323 )324 325# Manual testing of cat reading.326if 0 or test_all:327 Framework.File_Manager.File_System.Delayed_Init()328 # Test: open up a cat file, the one with text pages.329 cat09 = Framework.File_Manager.Cat_Reader.Cat_Reader(330 Settings.path_to_x4_folder / '09.cat')331 332 # Read the files from it.333 t44 = cat09.Read('t/0001-L044.xml')334 335 # Now try out the source reader.336 reader = Framework.File_Manager.Source_Reader.Source_Reader_class()337 reader.Init_From_Settings()338 t44_game_file = reader.Read('t/0001-L044.xml')339 jobs_game_file = reader.Read('libraries/jobs.xml')340 341 # Write to a new cat file.342 binary = t44_game_file.Get_Binary()343 cat_dir = Settings.path_to_x4_folder / 'extensions' / 'test_mod'344 if not cat_dir.exists():345 cat_dir.mkdir(parents = True)346 347 cat_writer = Framework.File_Manager.Cat_Writer.Cat_Writer(348 cat_path = cat_dir / 'test_01.cat')349 cat_writer.Add_File(t44_game_file)350 cat_writer.Write()351if 0 or test_all:352 Write_To_Extension()...
context.py
Source:context.py
...141 # '\tIs PN'142 '\n' )143 144 @staticmethod145 def test_all(tests):146 for t in tests:147 if not t:148 Context.test_fail()149 return150 Context.test_pass()151 @staticmethod152 def test_any(tests):153 for t in tests:154 if t:155 Context.test_pass()156 return157 Context.test_fail()158 159 @staticmethod160 def test_pass():161 stdout.write( '\t{}'.format(1) )162 @staticmethod163 def test_fail():164 stdout.write( '\t{}'.format(0) )165 166 @staticmethod167 def write(line, index, word, args):168 # Variables that we may use as part of other features.169 lemmata = line.get_lemmata(word)170 signs = word.split('-')171 (leftcx, leftlem) = \172 Context.get_left_context(line, index, word, args)173 (leftcx2, leftlem2) = \174 Context.get_left_context(line, index, word, args, offset = 2)175 (rightcx, rightlem) = \176 Context.get_right_context(line, index, word, args)177 # Print raw lemma tag. May include gloss, even when178 # --nogloss switch is provided. This is for the benefit179 # of human readers with some familiarity with Sumerian.180 181 stdout.write( '\t"{}/{}"'.format( word,182 Context.format_context(lemmata) ))183 # Index of word in line. 0-based.184 stdout.write( '\t{}'.format(index) )185 # Left context.186 stdout.write( '\t{}'.format(leftcx) )187 # Right context.188 stdout.write( '\t{}'.format(rightcx) )189 # Print line context.190 stdout.write( '\t"{}"'.format(line.line) )191 # Is word alone on line ?192 Context.test_all([ (leftcx, rightcx) == (None, None) ])193 194 # Left context is dumu.195 Context.test_all([ (leftcx == 'dumu') ])196 # Right context is dumu.197 Context.test_all([ (rightcx == 'dumu') ])198 # ^ ki <word> $199 Context.test_all([ (leftcx2, leftcx, rightcx) == \200 (None, 'ki', None) ])201 202 # ^ igi <word> $203 Context.test_all([ (leftcx2, leftcx, rightcx) == \204 (None, 'igi', None) ])205 # ^ igi <word>-sze $206 Context.test_all([ (leftcx2, leftcx, rightcx) == (None, 'igi', None),207 word.endswith('-sze3') ])208 # Personnenkeil: ^ 1(disz) <word> $209 Context.test_all([ (leftcx2, leftcx, rightcx) == \210 (None, '1(disz)', None) ])211 # ^ kiszib3 <word> $212 Context.test_all([ (leftcx2, leftcx, rightcx) == \213 (None, 'kiszib3', None) ])214 # ^ giri3 <word> $215 Context.test_all([ (leftcx2, leftcx, rightcx) == \216 (None, 'giri3', None) ])217 # First syllable repeated218 if len(signs) > 1:219 Context.test_all([ signs[0] == signs[1] ])220 else:221 Context.test_fail()222 # Last syllable repeated223 signs = word.split('-')224 if len(signs) > 1:225 Context.test_all([ signs[-2] == signs[-1] ])226 else:227 Context.test_fail()228 # Any syllable repeated229 if len(signs) > 1:230 Context.test_any([ a == b for (a, b)231 in zip(signs, signs[1:]) ])232 else:233 Context.test_fail()234 # Is profession235 Context.test_any([ pf == lem236 for pf in Context.professions237 for lem in lemmata ])238 # Contains profession239 Context.test_any([ pf in lem 240 for pf in Context.professions241 for lem in lemmata ])242 # Left context is profession243 if leftlem:244 Context.test_any([ pf == lem245 for pf in Context.professions246 for lem in line.get_lemmata(leftcx) ])247 else:248 Context.test_fail()249 # Left context contains profession250 if leftlem:251 Context.test_any([ pf in lem252 for pf in Context.professions253 for lem in line.get_lemmata(leftcx) ])254 else:255 Context.test_fail()256 # Right context is profession257 if rightlem:258 Context.test_any([ pf == lem259 for pf in Context.professions260 for lem in line.get_lemmata(rightcx) ])261 else:262 Context.test_fail()263 # Right context contains profession264 if rightlem:265 Context.test_any([ pf in lem266 for pf in Context.professions267 for lem in line.get_lemmata(rightcx) ])268 else:269 Context.test_fail()270 # Starts with ur-271 Context.test_all([ word.startswith('ur-') ])272 273 # Starts with lu2-274 Context.test_all([ word.startswith('lu2-') ])275 276 # Ends with -mu277 Context.test_all([ word.endswith('-mu') ])278 279 # Contains {d}280 Context.test_all([ '{d}' in word ])281 # Contains {ki}282 Context.test_all([ '{ki}' in word ])283 # Contains any determinative284 Context.test_all([ '{' in word ])285 # Contains q sound286 Context.test_all([ 'q' in word ])287 # Contains lugal288 Context.test_all([ 'lugal' in word ])289 # Contains numeric elements290 Context.test_any([ '(asz)' in word,291 '(disz)' in word,292 '(u)' in word ])293 # Followed by sag294 Context.test_all([ rightcx == 'sag' ])295 # Followed by zarin296 Context.test_all([ rightcx == 'zarin' ])297 # Preceded by numeric classifier298 Context.test_all([ leftcx in ( 'ba-an', 'ba-ri2-ga', 'bur3', 'da-na',299 'gin2-tur', 'gin2', 'gur-lugal', 300 'gur-sag-gal2', 'gur', 'iku', 'GAN2',301 'ku-li-mu', 'ku-li-kam', 'kusz3',302 'sar', 'sila3' ) ])303 # iti at head of sentence304 Context.test_all([ 'iti' == line.words[0][0] ])305 # mu at head of sentence306 Context.test_all([ 'mu' == line.words[0][0] ])307 """308 # Print boolean feature, 1 if word is PN, 0 if not.309 if 'PN' in lemmata:310 Context.test_pass()311 else:312 Context.test_fail()313 # Print most common tag for this word.314 # Actually, don't print the most common tag for this word.315 # This is not a good feature for CRF (or any feature-based316 # NLP algorithm) since it's # not immediately calculable317 # from context as a feature should be, but rather requires318 # reference to the index of the entire corpus.319 if word in INDEX:320 (bestlem, _) = INDEX[word].most_common(1)[0]...
Automation Testing Tutorials
Learn to execute automation testing from scratch with LambdaTest Learning Hub. Right from setting up the prerequisites to run your first automation test, to following best practices and diving deeper into advanced test scenarios. LambdaTest Learning Hubs compile a list of step-by-step guides to help you be proficient with different test automation frameworks i.e. Selenium, Cypress, TestNG etc.
LambdaTest Learning Hubs:
- JUnit Tutorial
- TestNG Tutorial
- Webdriver Tutorial
- WebDriverIO Tutorial
- Protractor Tutorial
- Selenium 4 Tutorial
- Jenkins Tutorial
- NUnit Tutorial
- Jest Tutorial
- Playwright Tutorial
- Cypress Tutorial
- PyTest Tutorial
YouTube
You could also refer to video tutorials over LambdaTest YouTube channel to get step by step demonstration from industry experts.
Run autotest automation tests on LambdaTest cloud grid
Perform automation testing on 3000+ real desktop and mobile devices online.
Try LambdaTest Now !!
Get 100 minutes of automation test minutes FREE!!
Was this article helpful?
