自动化测试工具 maven surefire 插件自定义输出,供失败重试作为物料

胡刚 · September 09, 2015 · 1170 hits

一.背景

在使用maven surefire插件命令:mvn test -Dtest=测试类进行测试时,对于stdout信息进行识别能迅速的发现问题,当某个suite类聚合了几千个用例时,失败数往往有几十甚至上百(虽然surefire支持运行中重试,但几千个用例同时运行时,同时申请资源,比如测试账号时,资源不够;或者并发量太大,测试环境web容器扛不住等),有必要进行迭代重试,而surefire生成的stdout Results结果没有成功和skipped用例,失败的用例信息格式多样,不便提取到有用信息;故修改surefire源码,统一格式输出。

标准的stdout如下:

这里写图片描述

二.解决办法

修改surefire源码,将执行结果输出进行统一格式输出,便于正则匹配,作为物料供重试使用(JUnitCore类)。
surefire github:https://github.com/apache/maven-surefire

1.下载源码

git clone https://github.com/apache/maven-surefire.git

2.修改源码

修改文件:

package org.apache.maven.plugin.surefire.report;

DefaultReporterFactory.java;

该文件使用工厂模式,监听执行结果,判断测试方法执行结果,放到各自map中,再输出,可以看注释分析。

package org.apache.maven.plugin.surefire.report;

/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/


import org.apache.maven.plugin.surefire.StartupReportConfiguration;
import org.apache.maven.plugin.surefire.runorder.StatisticsReporter;
import org.apache.maven.surefire.report.DefaultDirectConsoleReporter;
import org.apache.maven.surefire.report.ReporterFactory;
import org.apache.maven.surefire.report.RunListener;
import org.apache.maven.surefire.report.RunStatistics;
import org.apache.maven.surefire.report.StackTraceWriter;
import org.apache.maven.surefire.suite.RunResult;

import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.TreeMap;

/**
* Provides reporting modules on the plugin side.
* <p/>
* Keeps a centralized count of test run results.
*
* @author Kristian Rosenvold
*
* modify by hugang
* stdout success and skipped
*/

public class DefaultReporterFactory
implements ReporterFactory
{

private RunStatistics globalStats = new RunStatistics();

private final StartupReportConfiguration reportConfiguration;

private final StatisticsReporter statisticsReporter;

private final List<TestSetRunListener> listeners =
Collections.synchronizedList( new ArrayList<TestSetRunListener>() );

// from "<testclass>.<testmethod>" -> statistics about all the runs for flaky tests
private Map<String, List<TestMethodStats>> flakyTests;

// from "<testclass>.<testmethod>" -> statistics about all the runs for failed tests
private Map<String, List<TestMethodStats>> failedTests;

// from "<testclass>.<testmethod>" -> statistics about all the runs for error tests
private Map<String, List<TestMethodStats>> errorTests;

// added by hugang, from "<testclass>.<testmethod>" -> statistics about all the runs for success tests
private Map<String, List<TestMethodStats>> successTests;


// added by hugang, from "<testclass>.<testmethod>" -> statistics about all the runs for skipped tests
private Map<String, List<TestMethodStats>> skippedTests;




public DefaultReporterFactory( StartupReportConfiguration reportConfiguration )
{
this.reportConfiguration = reportConfiguration;
this.statisticsReporter = reportConfiguration.instantiateStatisticsReporter();
}

public RunListener createReporter()
{
return createTestSetRunListener();
}

public void mergeFromOtherFactories( Collection<DefaultReporterFactory> factories )
{
for ( DefaultReporterFactory factory : factories )
{
for ( TestSetRunListener listener : factory.listeners )
{
listeners.add( listener );
}
}
}

public RunListener createTestSetRunListener()
{
TestSetRunListener testSetRunListener =
new TestSetRunListener( reportConfiguration.instantiateConsoleReporter(),
reportConfiguration.instantiateFileReporter(),
reportConfiguration.instantiateStatelessXmlReporter(),
reportConfiguration.instantiateConsoleOutputFileReporter(), statisticsReporter,
reportConfiguration.isTrimStackTrace(),
ConsoleReporter.PLAIN.equals( reportConfiguration.getReportFormat() ),
reportConfiguration.isBriefOrPlainFormat() );
listeners.add( testSetRunListener );
return testSetRunListener;
}

public void addListener( TestSetRunListener listener )
{
listeners.add( listener );
}

public RunResult close()
{
mergeTestHistoryResult();
runCompleted();
for ( TestSetRunListener listener : listeners )
{
listener.close();
}
return globalStats.getRunResult();
}

private DefaultDirectConsoleReporter createConsoleLogger()
{
return new DefaultDirectConsoleReporter( reportConfiguration.getOriginalSystemOut() );
}

// 测试开始
public void runStarting()
{
final DefaultDirectConsoleReporter consoleReporter = createConsoleLogger();
consoleReporter.info( "" );
consoleReporter.info( "-------------------------------------------------------" );
consoleReporter.info( " T E S T S" );
consoleReporter.info( "-------------------------------------------------------" );
}

// 测试结束
private void runCompleted()
{
final DefaultDirectConsoleReporter logger = createConsoleLogger();
if ( reportConfiguration.isPrintSummary() )
{
logger.info( "" );
logger.info( "Results :" );
logger.info( "" );
}
// 输出不同类型用例信息
boolean printedFailures = printTestFailures( logger, TestResultType.failure );
printedFailures |= printTestFailures( logger, TestResultType.error );
printedFailures |= printTestFailures( logger, TestResultType.flake );


// added by hugang, 添加success and skipped用例详细
boolean printedSuccessSkipped = printTestSuccessSkipped( logger, TestResultType.success );
printedSuccessSkipped |= printTestSuccessSkipped( logger, TestResultType.skipped );

if ( printedFailures )
{
logger.info( "" );
}

if ( printedSuccessSkipped )
{
logger.info( "" );
}

// 输出failed 和 error的用例集, 作为下次重跑物料
// private Map<String, List<TestMethodStats>> failedTests;
// private Map<String, List<TestMethodStats>> errorTests;
// logger.info( failedTests.toString() );
// logger.info( "" );
// logger.info( errorTests.toString() );


// globalStats.getSummary() 打印数量
logger.info( globalStats.getSummary() );
logger.info( "" );
}

public RunStatistics getGlobalRunStatistics()
{
mergeTestHistoryResult();
return globalStats;
}

public static DefaultReporterFactory defaultNoXml()
{
return new DefaultReporterFactory( StartupReportConfiguration.defaultNoXml() );
}

/**
* Get the result of a test based on all its runs. If it has success and failures/errors, then it is a flake;
* if it only has errors or failures, then count its result based on its first run
*
* @param reportEntryList the list of test run report type for a given test
* @param rerunFailingTestsCount configured rerun count for failing tests
* @return the type of test result
*/

// Use default visibility for testing
static TestResultType getTestResultType( List<ReportEntryType> reportEntryList, int rerunFailingTestsCount )
{
if ( reportEntryList == null || reportEntryList.isEmpty() )
{
return TestResultType.unknown;
}

boolean seenSuccess = false, seenFailure = false, seenError = false;
for ( ReportEntryType resultType : reportEntryList )
{
if ( resultType == ReportEntryType.SUCCESS )
{
seenSuccess = true;
}
else if ( resultType == ReportEntryType.FAILURE )
{
seenFailure = true;
}
else if ( resultType == ReportEntryType.ERROR )
{
seenError = true;
}
}

if ( seenFailure || seenError )
{
if ( seenSuccess && rerunFailingTestsCount > 0 )
{
return TestResultType.flake;
}
else
{
if ( seenError )
{
return TestResultType.error;
}
else
{
return TestResultType.failure;
}
}
}
else if ( seenSuccess )
{
return TestResultType.success;
}
else
{
return TestResultType.skipped;
}
}

/**
* Merge all the TestMethodStats in each TestRunListeners and put results into flakyTests, failedTests and
* errorTests, indexed by test class and method name. Update globalStatistics based on the result of the merge.
*/

void mergeTestHistoryResult()
{
globalStats = new RunStatistics();
flakyTests = new TreeMap<String, List<TestMethodStats>>();
failedTests = new TreeMap<String, List<TestMethodStats>>();
errorTests = new TreeMap<String, List<TestMethodStats>>();
// added by hugang, 存success 和 skpped 用例信息
successTests = new TreeMap<String, List<TestMethodStats>>();
skippedTests = new TreeMap<String, List<TestMethodStats>>();

Map<String, List<TestMethodStats>> mergedTestHistoryResult = new HashMap<String, List<TestMethodStats>>();
// Merge all the stats for tests from listeners
for ( TestSetRunListener listener : listeners )
{
List<TestMethodStats> testMethodStats = listener.getTestMethodStats();
for ( TestMethodStats methodStats : testMethodStats )
{
List<TestMethodStats> currentMethodStats =
mergedTestHistoryResult.get( methodStats.getTestClassMethodName() );
if ( currentMethodStats == null )
{
currentMethodStats = new ArrayList<TestMethodStats>();
currentMethodStats.add( methodStats );
mergedTestHistoryResult.put( methodStats.getTestClassMethodName(), currentMethodStats );
}
else
{
currentMethodStats.add( methodStats );
}
}
}

// Update globalStatistics by iterating through mergedTestHistoryResult
int completedCount = 0, skipped = 0;
// 遍历所有的类,判断每一个类中的方法执行结果,放到对应的map中;
// TestMethodStats每个测试方法信息
for ( Map.Entry<String, List<TestMethodStats>> entry : mergedTestHistoryResult.entrySet() )
{
List<TestMethodStats> testMethodStats = entry.getValue();
String testClassMethodName = entry.getKey();
completedCount++;

// 将每个测试方法的执行结果添加到resultTypeList中
List<ReportEntryType> resultTypeList = new ArrayList<ReportEntryType>();
for ( TestMethodStats methodStats : testMethodStats )
{
resultTypeList.add( methodStats.getResultType() );
}

TestResultType resultType = getTestResultType( resultTypeList,
reportConfiguration.getRerunFailingTestsCount() );
// 根据一个类的不同方法执行结果,放到对应的map中
switch ( resultType )
{
case success:
// If there are multiple successful runs of the same test, count all of them
int successCount = 0;
for ( ReportEntryType type : resultTypeList )
{
if ( type == ReportEntryType.SUCCESS )
{
successCount++;
}
}
completedCount += successCount - 1;
// added by hugang, 把success 用例信息,put 到map中
successTests.put( testClassMethodName, testMethodStats );
break;
case skipped:
// added by hugang, 把skipped 用例信息,put 到map中
skippedTests.put( testClassMethodName, testMethodStats );
skipped++;
break;
case flake:
flakyTests.put( testClassMethodName, testMethodStats );
break;
case failure:
failedTests.put( testClassMethodName, testMethodStats );
break;
case error:
errorTests.put( testClassMethodName, testMethodStats );
break;
default:
throw new IllegalStateException( "Get unknown test result type" );
}
}

globalStats.set( completedCount, errorTests.size(), failedTests.size(), skipped, flakyTests.size() );
}

/**
* Print failed tests and flaked tests. A test is considered as a failed test if it failed/got an error with
* all the runs. If a test passes in ever of the reruns, it will be count as a flaked test
*
* @param logger the logger used to log information
* @param type the type of results to be printed, could be error, failure or flake
* @return {@code true} if printed some lines
*/

private String statckInfo = "WeiboQA failed cases StackTrace";
// Use default visibility for testing
boolean printTestFailures( DefaultDirectConsoleReporter logger, TestResultType type )
{
boolean printed = false;
final Map<String, List<TestMethodStats>> testStats;
switch ( type )
{
case failure:
testStats = failedTests;
break;
case error:
testStats = errorTests;
break;
case flake:
testStats = flakyTests;
break;
default:
return printed;
}

if ( !testStats.isEmpty() )
{
// 被注释,添加到每行用例信息前,便于正则匹配
// logger.info( type.getLogPrefix() );
printed = true;
}

for ( Map.Entry<String, List<TestMethodStats>> entry : testStats.entrySet() )
{
printed = true;
List<TestMethodStats> testMethodStats = entry.getValue();
if ( testMethodStats.size() == 1 )
{
// 被注释
// No rerun, follow the original output format
// logger.info( " " + testMethodStats.get( 0 ).getStackTraceWriter().smartTrimmedStackTrace() );
// added by hugang , 每行用例信息前,便于正则匹配
// 打印失败信息
logger.info( statckInfo + "---"
+ testMethodStats.get( 0 ).getStackTraceWriter().smartTrimmedStackTrace() );
// 只打印失败的类方法
logger.info( type.getLogPrefix() + "---"
+ testMethodStats.get( 0 ).getTestClassMethodName() );
}
else
{
logger.info( statckInfo + "---" + entry.getKey() );
for ( int i = 0; i < testMethodStats.size(); i++ )
{
StackTraceWriter failureStackTrace = testMethodStats.get( i ).getStackTraceWriter();
if ( failureStackTrace == null )
{
logger.info( " Run " + ( i + 1 ) + ": PASS" );
}
else
{
logger.info( " Run " + ( i + 1 ) + ": " + failureStackTrace.smartTrimmedStackTrace() );
}
}
// 只打印失败的类方法
logger.info( type.getLogPrefix() + "---"
+ testMethodStats.get( 0 ).getTestClassMethodName() );
logger.info( "" );
}
}
return printed;
}

// 打印成功和skipped用例
boolean printTestSuccessSkipped( DefaultDirectConsoleReporter logger, TestResultType type )
{
boolean printed = false;
final Map<String, List<TestMethodStats>> testStats;
switch ( type )
{
// added by hugang;支持success and skipped
case success:
testStats = successTests;
break;
case skipped:
testStats = skippedTests;
break;
default:
return printed;
}

if ( !testStats.isEmpty() )
{
// 被注释,添加到每行用例信息前,便于正则匹配
// logger.info( type.getLogPrefix() );
printed = true;
}

for ( Map.Entry<String, List<TestMethodStats>> entry : testStats.entrySet() )
{
printed = true;
List<TestMethodStats> testMethodStats = entry.getValue();
if ( testMethodStats.size() == 1 )
{
// 被注释
// No rerun, follow the original output format
// logger.info( " " + testMethodStats.get( 0 ).getStackTraceWriter().smartTrimmedStackTrace() );
// added by hugang , 每行用例信息前,便于正则匹配
logger.info( type.getLogPrefix() + "---" + testMethodStats.get( 0 ).getTestClassMethodName() );
}
else
{
logger.info( entry.getKey() );
for ( int i = 0; i < testMethodStats.size(); i++ )
{
logger.info( type.getLogPrefix() + "---" + testMethodStats.get( i ).getTestClassMethodName() );
}
logger.info( "" );
}
}
return printed;
}
// Describe the result of a given test
static enum TestResultType
{

error( "WeiboQA Test Error info: " ),
failure( "WeiboQA Test Fail info: " ),
flake( "WeiboQA Test Flaked info: " ),
success( "WeiboQA Test Success info: " ),
skipped( "WeiboQA Test Skipped info: " ),
unknown( "WeiboQA Test Unknown info: " );

private final String logPrefix;

private TestResultType( String logPrefix )
{
this.logPrefix = logPrefix;
}

public String getLogPrefix()
{
return logPrefix;
}
}
}

3.本地安装自定义surefire插件

3-1. 进入surefire-surefire目录下
hugangdeMacBook-Pro:maven-surefire root# pwd
/Users/hugang/myworkspace/maven-surefire

hugangdeMacBook-Pro:maven-surefire root# mvn clean -Dmaven.test.skip=true install
3-2.进入maven-surefire/maven-surefire-plugin目录下
hugangdeMacBook-Pro:maven-surefire-plugin root# pwd
/Users/hugang/myworkspace/maven-surefire/maven-surefire-plugin

hugangdeMacBook-Pro:maven-surefire-plugin root# mvn clean -Dmaven.test.skip=true install
3-3.进入maven-surefire/maven-surefire-common
hugangdeMacBook-Pro:maven-surefire-common root# pwd
/Users/hugang/myworkspace/maven-surefire/maven-surefire-common

hugangdeMacBook-Pro:maven-surefire-common root# mvn clean -Dmaven.test.skip=true install

4.在测试项目中,引入自定义surefire插件

在pom.xml中引用自定义插件(自定义surefire插件版本为2.19-SNAPSHOT):

<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<!-- <version>2.18.1</version> -->
<version>2.19-SNAPSHOT</version>
<configuration>
<!-- <forkMode>pertest</forkMode>
<argLine>-Xms1024m -Xmx1024m</argLine>
--> <printSummary>true</printSummary>
<testFailureIgnore>true</testFailureIgnore>
</configuration>
<dependencies>
<dependency>
<groupId>org.apache.maven.surefire</groupId>
<artifactId>surefire-junit47</artifactId>
<version>2.18.1</version>
</dependency>
</dependencies>
</plugin>
</plugins>

5.mvn test执行用例

mvn test -Dtest=WangleiTestSuite

stdout输出结果如下图:

这里写图片描述

错误用例失败信息,用于分析失败原因:

前缀为WeiboQA failed cases StackTrace

统计成功用例信息,可以解析出具体的测试方法:

WeiboQA Test Success info:  测试方法(测试类)

统计skipped用例信息:

WeiboQA Test Skipped info: 测试方法(测试类)

统计error用例信息,供失败重试,作为物料,进行重试:

WeiboQA Test Error info: 测试方法(测试类)

统计failed用例信息,供失败重试作为物料,进行重试:

WeiboQA Test Fail info: 测试方法(测试类)

6.重试失败用例

只需匹配:

^WeiboQA Test Error info:
^WeiboQA Test Fail info:

将错误方法和对应的类找出,使用JUnitCore重新执行,具体可以参考: JUnit结果重跑失败用例(支持Mvn和Ant),修改正则表达式即可。

如果觉得我的文章对您有用,请随意打赏。您的支持将鼓励我继续创作!
No Reply at the moment.
需要 Sign In 后方可回复, 如果你还没有账号请点击这里 Sign Up